[CARBONDATA-4322] Apply local sort task level property for insert
authorShreelekhyaG <shreelu_gampa@yahoo.com>
Mon, 24 Jan 2022 12:37:19 +0000 (18:07 +0530)
committerIndhumathi27 <indhumathim27@gmail.com>
Mon, 14 Feb 2022 15:55:35 +0000 (21:25 +0530)
commit59f23c0dfea74199f97b6d3626c31d8fba4a2e1f
tree603f6139265923754ea44fa5a09be0b561494030
parent05aff876d4e7ae7dcea2cecda176b470eb658ff8
[CARBONDATA-4322] Apply local sort task level property for insert

Why is this PR needed?
Currently, When carbon.partition.data.on.tasklevel is enabled with
local sort, the number of tasks launched for load will be based on
node locality. But for insert command, the local sort task level
property is not applied which is causing the number of tasks
launched based on the input files.

What changes were proposed in this PR?
Included changes to apply carbon.partition.data.on.tasklevel property
for insert command as well. Used DataLoadCoalescedRDD to coalesce
the partitions and a DataLoadCoalescedUnwrapRDDto unwrap partitions
from DataLoadPartitionWrap and iterate.

Does this PR introduce any user interface change?
No

Is any new testcase added?
Yes

This closes #4248
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CommonLoadUtils.scala
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/standardpartition/StandardPartitionTableCompactionTestCase.scala