[CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
authorShreelekhyaG <shreelu_gampa@yahoo.com>
Wed, 29 Sep 2021 15:48:13 +0000 (21:18 +0530)
committerIndhumathi27 <indhumathim27@gmail.com>
Thu, 21 Oct 2021 15:07:45 +0000 (20:37 +0530)
commit305851ed75cf935c2a606071118dbe1347a18628
treef03ddd0bc73ceda7fb85b6004059c759dde57cb1
parentb8d9a97926db77a8d75c8988b88c9d5f74df3415
[CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type

Why is this PR needed?
1. IS_EMPTY_DATA_BAD_RECORD property not supported for complex types.
2. To update documentation that COLUMN_META_CACHE and RANGE_COLUMN
   doesn't support complex datatype

What changes were proposed in this PR?
1. Made changes to pass down IS_EMPTY_DATA_BAD_RECORD property and
   throw exception. Store empty complex type instead of storing
   null value which matches with hive table result.
2. Updated document and added testcase.

Does this PR introduce any user interface change?
No

Is any new testcase added?
Yes

This closes #4228
18 files changed:
core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
docs/ddl-of-carbondata.md
integration/spark/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
integration/spark/src/test/resources/complextypeWithEmptyRecords.csv [new file with mode: 0644]
integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/badrecordloger/BadRecordEmptyDataTest.scala
integration/spark/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOpName.scala
processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java
processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java
processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java
processing/src/main/java/org/apache/carbondata/processing/datatypes/StructDataType.java
processing/src/main/java/org/apache/carbondata/processing/loading/converter/impl/ComplexFieldConverterImpl.java
processing/src/main/java/org/apache/carbondata/processing/loading/converter/impl/FieldEncoderFactory.java
processing/src/main/java/org/apache/carbondata/processing/loading/parser/impl/ArrayParserImpl.java
processing/src/main/java/org/apache/carbondata/processing/loading/parser/impl/MapParserImpl.java
processing/src/main/java/org/apache/carbondata/processing/loading/parser/impl/StructParserImpl.java
processing/src/main/java/org/apache/carbondata/processing/loading/steps/InputProcessorStepWithNoConverterImpl.java
processing/src/main/java/org/apache/carbondata/processing/util/CarbonBadRecordUtil.java