[CARBONDATA-4247][CARBONDATA-4241] Fix Wrong timestamp value query results for data...
authorIndhumathi27 <indhumathim27@gmail.com>
Tue, 20 Jul 2021 08:12:57 +0000 (13:42 +0530)
committerakashrn5 <akashnilugal@gmail.com>
Thu, 29 Jul 2021 11:18:18 +0000 (16:48 +0530)
commitfeb052173b5c834dfa3632b01df275ed061aea1d
treee8dd166e8aefeb85cb84da8f1df9001ebe90737c
parentf2698fed9b83fbdca5cc473ffac1ebcdd7880e23
[CARBONDATA-4247][CARBONDATA-4241] Fix Wrong timestamp value query results for data before
1900 years with Spark 3.1

Why is this PR needed?
1. Spark 3.1, will store timestamp value as julian micros and rebase timestamp value from
JulianToGregorianMicros during query.
-> Since carbon parse and formats timestamp value with SimpleDateFormatter, query gives
incorrect results, when rebased with JulianToGregorianMicros by spark.
2. CARBONDATA-4241 -> Global sort load and compaction fails on table having timestamp column

What changes were proposed in this PR?
1. Use Java Instant to parse new timestamp values. For old stores and query with Spark 3.1,
Rebase the timestamp value from Julian to Gregorian Micros
2. If timestamp value is of type Instant, then convert value to java timestamp.

Does this PR introduce any user interface change?
No

Is any new testcase added?
No (Existing testcase is sufficient)

This closes #4177
34 files changed:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
core/src/main/java/org/apache/carbondata/core/datastore/block/AbstractIndex.java
core/src/main/java/org/apache/carbondata/core/datastore/block/TableBlockInfo.java
core/src/main/java/org/apache/carbondata/core/datastore/chunk/reader/dimension/v3/DimensionChunkReaderV3.java
core/src/main/java/org/apache/carbondata/core/datastore/chunk/reader/measure/v3/MeasureChunkReaderV3.java
core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/IndexWrapper.java
core/src/main/java/org/apache/carbondata/core/metadata/blocklet/DataFileFooter.java
core/src/main/java/org/apache/carbondata/core/scan/collector/impl/DictionaryBasedVectorResultCollector.java
core/src/main/java/org/apache/carbondata/core/scan/executor/impl/AbstractQueryExecutor.java
core/src/main/java/org/apache/carbondata/core/scan/processor/DataBlockIterator.java
core/src/main/java/org/apache/carbondata/core/scan/result/RowBatch.java
core/src/main/java/org/apache/carbondata/core/scan/result/iterator/AbstractDetailQueryResultIterator.java
core/src/main/java/org/apache/carbondata/core/scan/result/iterator/ChunkRowIterator.java
core/src/main/java/org/apache/carbondata/core/scan/result/iterator/DetailQueryResultIterator.java
core/src/main/java/org/apache/carbondata/core/scan/result/vector/CarbonColumnVector.java
core/src/main/java/org/apache/carbondata/core/scan/result/vector/CarbonColumnarBatch.java
core/src/main/java/org/apache/carbondata/core/scan/result/vector/ColumnVectorInfo.java
core/src/main/java/org/apache/carbondata/core/scan/result/vector/impl/CarbonColumnVectorImpl.java
core/src/main/java/org/apache/carbondata/core/scan/result/vector/impl/directread/ColumnarVectorWrapperDirectWithDeleteDelta.java
core/src/main/java/org/apache/carbondata/core/scan/result/vector/impl/directread/ColumnarVectorWrapperDirectWithInvertedIndex.java
core/src/main/java/org/apache/carbondata/core/util/DataFileFooterConverterV3.java
core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java
hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java
integration/presto/src/main/java/org/apache/carbondata/presto/ColumnarVectorWrapperDirect.java
integration/spark/src/main/common2.3and2.4/org/apache/spark/sql/SparkVersionAdapter.scala
integration/spark/src/main/java/org/apache/carbondata/spark/vectorreader/ColumnarVectorWrapper.java
integration/spark/src/main/java/org/apache/carbondata/spark/vectorreader/ColumnarVectorWrapperDirect.java
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonScanRDD.scala
integration/spark/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
integration/spark/src/main/spark3.1/org/apache/spark/sql/SparkVersionAdapter.scala
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/directdictionary/TimestampNoDictionaryColumnTestCase.scala
streaming/src/main/scala/org/apache/carbondata/streaming/parser/FieldConverter.scala
streaming/src/main/spark2.x/org.apache.carbondata.util/SparkStreamingUtil.scala
streaming/src/main/spark3.1/org/apache/carbondata/util/SparkStreamingUtil.scala