[CARBONDATA-4271] Support DPP for carbon
authorIndhumathi27 <indhumathim27@gmail.com>
Tue, 13 Jul 2021 11:37:39 +0000 (17:07 +0530)
committerkunal642 <kunalkapoor642@gmail.com>
Wed, 1 Sep 2021 06:12:23 +0000 (11:42 +0530)
commitbdc9484ac8455e8f53e86367c0e5104364799068
tree227f708fd3af05a8d646f2f3e64025e567ab1432
parentca659b5cd3219de7c8a9603784559e20fdff1fda
[CARBONDATA-4271] Support DPP for carbon

Why is this PR needed?
This PR enables Dynamic Partition Pruning for carbon.

What changes were proposed in this PR?
CarbonDatasourceHadoopRelation has to extend HadoopFsRelation,
because spark has added a check to use DPP only for relation matching HadoopFsRelation
Apply Dynamic filter and get runtimePartitions and set this to CarbonScanRDD for pruning

This closes #4199
17 files changed:
integration/spark/src/main/common2.3and2.4/org/apache/spark/sql/CarbonDataSourceScanHelper.scala
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDeltaRowScanRDD.scala
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonScanRDD.scala
integration/spark/src/main/scala/org/apache/spark/sql/CarbonDatasourceHadoopRelation.scala
integration/spark/src/main/scala/org/apache/spark/sql/CarbonSource.scala
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoHadoopFsRelationCommand.scala
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonCreateTableAsSelectCommand.scala
integration/spark/src/main/scala/org/apache/spark/sql/execution/strategy/CarbonDataSourceScan.scala
integration/spark/src/main/scala/org/apache/spark/sql/execution/strategy/CarbonPlanHelper.scala
integration/spark/src/main/scala/org/apache/spark/sql/execution/strategy/CarbonSourceStrategy.scala
integration/spark/src/main/scala/org/apache/spark/sql/execution/strategy/DMLStrategy.scala
integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
integration/spark/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala
integration/spark/src/main/spark3.1/org/apache/spark/sql/CarbonDataSourceScanHelper.scala
integration/spark/src/main/spark3.1/org/apache/spark/sql/CarbonToSparkAdapter.scala
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/dblocation/DBLocationCarbonTableTestCase.scala
integration/spark/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningTestCase.scala [new file with mode: 0644]