Display row stats for multiphase parallel indexing tasks (#12280)
authorTejaswini Bandlamudi <96047043+tejaswini-imply@users.noreply.github.com>
Wed, 2 Mar 2022 04:40:31 +0000 (10:10 +0530)
committerGitHub <noreply@github.com>
Wed, 2 Mar 2022 04:40:31 +0000 (10:10 +0530)
commit1af4c9c933cd62804d106c9a2974dc0b3cf75780
tree13089e6118152b72fc507ecbc727f6cfae6065b5
parent50038d9344fb745bfe47c81a59bdc29ae3dcff1d
Display row stats for multiphase parallel indexing tasks (#12280)

Row stats are reported for single phase tasks in the `/liveReports` and `/rowStats` APIs
and are also a part of the overall task report. This commit adds changes to report
row stats for multiphase tasks too.

Changes:
- Add `TaskReport` in `GeneratedPartitionsReport` generated during hash and range partitioning
- Collect the reports for `index_generate` phase in `ParallelIndexSupervisorTask`
14 files changed:
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/GeneratedPartitionsMetadataReport.java
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/GeneratedPartitionsReport.java
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/PartialHashSegmentGenerateTask.java
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/PartialRangeSegmentGenerateTask.java
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/PartialSegmentGenerateTask.java
indexing-service/src/test/java/org/apache/druid/indexing/common/task/batch/parallel/AbstractMultiPhaseParallelIndexingTest.java
indexing-service/src/test/java/org/apache/druid/indexing/common/task/batch/parallel/AbstractParallelIndexSupervisorTaskTest.java
indexing-service/src/test/java/org/apache/druid/indexing/common/task/batch/parallel/HashPartitionMultiPhaseParallelIndexingTest.java
indexing-service/src/test/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTaskTest.java
indexing-service/src/test/java/org/apache/druid/indexing/common/task/batch/parallel/RangePartitionMultiPhaseParallelIndexingTest.java
indexing-service/src/test/java/org/apache/druid/indexing/common/task/batch/parallel/SinglePhaseParallelIndexingTest.java
processing/src/main/java/org/apache/druid/segment/incremental/MutableRowIngestionMeters.java [new file with mode: 0644]
processing/src/test/java/org/apache/druid/segment/incremental/MutableRowIngestionMetersTest.java [new file with mode: 0644]