[CARBONDATA-4285] Fix alter add complex columns with global sort compaction failure
authorMahesh Raju Somalaraju <mahesh.somalaraju@huawei.com>
Thu, 16 Sep 2021 17:11:54 +0000 (22:41 +0530)
committerkunal642 <kunalkapoor642@gmail.com>
Mon, 20 Sep 2021 04:45:07 +0000 (10:15 +0530)
commit22342f847d7db515e5f8c17525522085f49bd2a5
tree37b118f1a37873f18c11e6641b4d00b2c81dcade
parent2d1907b912e0d9923e0a8fcae5f3dddc98fe5134
[CARBONDATA-4285] Fix alter add complex columns with global sort compaction failure

Why is this PR needed?
Alter add complex columns with global sort compaction is failing due to

AOI exception : Currently creating default complex delimiter list in global sort compaction
with size of 3. For map case need extra complex delimiter for handling the key-value
bad record handling: When we add complex columns after insert the data, complex columns
has null data for previously loaded segments. this null value is going to treat as bad
record and compaction is failed.

What changes were proposed in this PR?
In Global sort compaction flow create default complex delimiter with 4, as already
doing in load flow.
Bad records handling pruned for compaction case. No need to check bad records for
compaction as they are already checked while loading. previously loaded segments data
we are inserting again in compaction case

This closes #4218
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java
integration/spark/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
integration/spark/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessorStepOnSpark.scala
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/alterTable/TestAlterTableAddColumns.scala
processing/src/main/java/org/apache/carbondata/processing/loading/BadRecordsLogger.java
processing/src/main/java/org/apache/carbondata/processing/loading/BadRecordsLoggerProvider.java
processing/src/main/java/org/apache/carbondata/processing/loading/converter/impl/RowConverterImpl.java