hudi.git
10 months ago[HUDI-2801] Add Amazon CloudWatch metrics reporter (#4081)
Udit Mehrotra [Thu, 25 Nov 2021 21:33:16 +0000 (13:33 -0800)] 
[HUDI-2801] Add Amazon CloudWatch metrics reporter (#4081)

10 months ago[HUDI-2841] Fixing lazy rollback for MOR with list based strategy (#4110)
Sivabalan Narayanan [Thu, 25 Nov 2021 21:06:04 +0000 (16:06 -0500)] 
[HUDI-2841] Fixing lazy rollback for MOR with list based strategy (#4110)

10 months ago[HUDI-2858] Fixing handling of cluster update reject exception in deltastreamer ...
Sivabalan Narayanan [Thu, 25 Nov 2021 19:34:07 +0000 (14:34 -0500)] 
[HUDI-2858] Fixing handling of cluster update reject exception in deltastreamer (#4120)

10 months ago[HUDI-2671] Making error -> warn logs from timeline server with concurrent writers...
Sivabalan Narayanan [Thu, 25 Nov 2021 19:21:32 +0000 (14:21 -0500)] 
[HUDI-2671] Making error -> warn logs from timeline server with concurrent writers for inconsistent state (#4088)

* Making error -> warn logs from timeline server with concurrent writers for inconsistent state

* Fixing bad request response exception for timeline out of sync

* Addressing feedback. removed write concurrency mode depedency

10 months ago[HUDI-2794] Guarding table service commits within a single lock to commit to both...
Sivabalan Narayanan [Thu, 25 Nov 2021 19:19:30 +0000 (14:19 -0500)] 
[HUDI-2794] Guarding table service commits within a single lock to commit to both data table and metadata table (#4037)

* Fixing a single lock to commit table services across metadata table and data table

* Addressing comments

* rebasing with master

10 months ago[HUDI-2800] Remove rdd.isEmpty() validation to prevent CreateHandle being called...
Sagar Sumit [Thu, 25 Nov 2021 18:16:36 +0000 (23:46 +0530)] 
[HUDI-2800] Remove rdd.isEmpty() validation to prevent CreateHandle being called twice (#4121)

10 months ago[HUDI-1290] fixing mysql debezium source (#4119)
satishm [Thu, 25 Nov 2021 16:26:59 +0000 (21:56 +0530)] 
[HUDI-1290] fixing mysql debezium source (#4119)

10 months ago[HUDI-2480] FileSlice after pending compaction-requested instant-time… (#3703)
Danny Chan [Thu, 25 Nov 2021 14:30:09 +0000 (22:30 +0800)] 
[HUDI-2480] FileSlice after pending compaction-requested instant-time… (#3703)

* [HUDI-2480] FileSlice after pending compaction-requested instant-time is ignored by MOR snapshot reader

* include file slice after a pending compaction for spark reader

Co-authored-by: garyli1019 <yanjia.gary.li@gmail.com>
10 months ago[HUDI-2855] Change the default value of 'PAYLOAD_CLASS_NAME' to 'DefaultHoodieRecordP...
董可伦 [Thu, 25 Nov 2021 11:17:38 +0000 (19:17 +0800)] 
[HUDI-2855] Change the default value of 'PAYLOAD_CLASS_NAME' to 'DefaultHoodieRecordPayload' (#4115)

10 months ago[HUDI-2792] Configure metadata payload consistency check (#4035)
Sivabalan Narayanan [Thu, 25 Nov 2021 02:56:31 +0000 (21:56 -0500)] 
[HUDI-2792] Configure metadata payload consistency check (#4035)

- Relax metadata payload consistency check to consider spark task failures with spurious deletes

10 months ago[HUDI-1290] Add Debezium Source for deltastreamer (#4063)
rmahindra123 [Thu, 25 Nov 2021 01:57:02 +0000 (17:57 -0800)] 
[HUDI-1290] Add Debezium Source for deltastreamer (#4063)

* add source for postgres debezium

* Add tests for debezium payload

* Fix test

* Fix test

* Add tests for debezium source

* Add tests for debezium source

* Fix schema for debezium

* Fix checkstyle issues

* Fix config issue for schema registry

* Add mysql source for debezium

* Fix checkstyle issues an tests

* Improve code for merging toasted values

* Improve code for merging toasted values

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
10 months ago[HUDI-1290] [RFC-39] Deltastreamer avro source for Debezium CDC (#4048)
rmahindra123 [Thu, 25 Nov 2021 01:31:34 +0000 (17:31 -0800)] 
[HUDI-1290] [RFC-39] Deltastreamer avro source for Debezium CDC (#4048)

* Add RFC entry for deltastreamer source for debezium

* Add RFC for debezium source

* Add RFC for debezium source

* Add RFC for debezium source

* fix hyperlink issue and rebase

* Update progress

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
10 months ago[MINOR] Fix build failure due to checkstyle issues (#4111)
Y Ethan Guo [Thu, 25 Nov 2021 01:17:46 +0000 (17:17 -0800)] 
[MINOR] Fix build failure due to checkstyle issues (#4111)

10 months ago[HUDI-2844][CLI] Fixing archived Timeline crashing if timeline contains REPLACE_COMMI...
Alexey Kudinkin [Thu, 25 Nov 2021 00:53:29 +0000 (16:53 -0800)] 
[HUDI-2844][CLI] Fixing archived Timeline crashing if timeline contains REPLACE_COMMIT (#4091)

10 months ago[HUDI-2853] Add JMX deps in hudi utilities and kafka connect bundles (#4108)
rmahindra123 [Thu, 25 Nov 2021 00:03:01 +0000 (16:03 -0800)] 
[HUDI-2853] Add JMX deps in hudi utilities and kafka connect bundles (#4108)

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
10 months ago[HUDI-2793] Fixing deltastreamer checkpoint fetch/copy over (#4034)
Sivabalan Narayanan [Wed, 24 Nov 2021 23:26:40 +0000 (18:26 -0500)] 
[HUDI-2793] Fixing deltastreamer checkpoint fetch/copy over (#4034)

- Removed the copy over logic in transaction utils. Deltastreamer will go back to previous commits and get the checkpoint value.

10 months ago[HUDI-2766] Cluster update strategy should not be fenced by write config (#4093)
Sagar Sumit [Wed, 24 Nov 2021 18:15:40 +0000 (23:45 +0530)] 
[HUDI-2766] Cluster update strategy should not be fenced by write config (#4093)

Fix pending clustering rollback test

10 months ago[HUDI-2788] Fixing issues w/ Z-order Layout Optimization (#4026)
Alexey Kudinkin [Wed, 24 Nov 2021 18:10:28 +0000 (10:10 -0800)] 
[HUDI-2788] Fixing issues w/ Z-order Layout Optimization (#4026)

* Simplyfying, tidying up

* Fixed packaging for `TestOptimizeTable`

* Cleaned up `HoodiFileIndex` file filtering seq;
Removed optimization manually reading Parquet table circumventing Spark

* Refactored `DataSkippingUtils`:
  - Fixed checks to validate all statistics cols are present
  - Fixed some predicates being constructed incorrectly
  - Rewrote comments for easier comprehension, added more notes
  - Tidying up

* Tidying up tests

* `lint`

* Fixing compilation

* `TestOptimizeTable` > `TestTableLayoutOptimization`;
Added assertions to test data skipping paths

* Fixed tests to properly hit data-skipping path

* Fixed pruned files candidates lookup seq to conservatively included all non-indexed files

* Added java-doc

* Fixed compilation

10 months ago[HUDI-2443] Hudi KVComparator for all HFile writer usages (#3889)
Manoj Govindassamy [Wed, 24 Nov 2021 18:05:36 +0000 (10:05 -0800)] 
[HUDI-2443] Hudi KVComparator for all HFile writer usages (#3889)

* [HUDI-2443] Hudi KVComparator for all HFile writer usages

- Hudi relies on custom class shading for Hbase's KeyValue.KVComparator to
  avoid versioning and class loading issues. There are few places which are
  still using the Hbase's comparator class directly and version upgrades
  would make them obsolete. Refactoring the HoodieKVComparator and making
  all HFile writer creation using the same shaded class.

* [HUDI-2443] Hudi KVComparator for all HFile writer usages

- Moving HoodieKVComparator from common.bootstrap.index to common.util

* [HUDI-2443] Hudi KVComparator for all HFile writer usages

- Retaining the old HoodieKVComparatorV2 for boostrap case. Adding the
  new comparator as HoodieKVComparatorV2 to differentiate from the old
  one.

* [HUDI-2443] Hudi KVComparator for all HFile writer usages

 - Renamed HoodieKVComparatorV2 to HoodieMetadataKVComparator and moved it
   under the package org.apache.hudi.metadata.

* Make comparator classname configurable

* Revert new config and address other review comments

Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
10 months ago[HUDI-2671] Fix kafka offset handling in Kafka Connect protocol (#4021)
rmahindra123 [Wed, 24 Nov 2021 18:03:58 +0000 (10:03 -0800)] 
[HUDI-2671] Fix kafka offset handling in Kafka Connect protocol (#4021)

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
10 months ago[HUDI-2688] Claim the next rfc 40 for Hudi connector for Trino (#4105)
Sagar Sumit [Wed, 24 Nov 2021 16:43:37 +0000 (22:13 +0530)] 
[HUDI-2688] Claim the next rfc 40 for Hudi connector for Trino (#4105)

10 months ago[HUDI-2759] extract HoodieCatalogTable to coordinate spark catalog table and hoodie...
Yann Byron [Wed, 24 Nov 2021 10:12:38 +0000 (18:12 +0800)] 
[HUDI-2759] extract HoodieCatalogTable to coordinate spark catalog table and hoodie table (#3998)

10 months ago[HUDI-2847] Flink metadata table supports virtual keys (#4096)
Danny Chan [Wed, 24 Nov 2021 09:34:42 +0000 (17:34 +0800)] 
[HUDI-2847] Flink metadata table supports virtual keys (#4096)

10 months agoRevert "[HUDI-2799] Fix the classloader of flink write task (#4042)" (#4069)
Danny Chan [Wed, 24 Nov 2021 04:01:18 +0000 (12:01 +0800)] 
Revert "[HUDI-2799] Fix the classloader of flink write task (#4042)" (#4069)

This reverts commit 8281cbf7624c3a4eb90bf58671daf76843d00819.

10 months ago[HUDI-2838] refresh table after drop partition (#4084)
Yann Byron [Wed, 24 Nov 2021 03:46:48 +0000 (11:46 +0800)] 
[HUDI-2838] refresh table after drop partition (#4084)

10 months ago[HUDI-2818] Fix 2to3 upgrade when set `hoodie.table.keygenerator.class` (#4077)
Raymond Xu [Wed, 24 Nov 2021 03:43:34 +0000 (19:43 -0800)] 
[HUDI-2818] Fix 2to3 upgrade when set `hoodie.table.keygenerator.class` (#4077)

10 months ago[HUDI-2831] Securing usages of `SimpleDateFormat` to be thread-safe (#4073)
Alexey Kudinkin [Wed, 24 Nov 2021 01:25:11 +0000 (17:25 -0800)] 
[HUDI-2831] Securing usages of `SimpleDateFormat` to be thread-safe (#4073)

10 months ago[HUDI-2325] Add hive sync support to kafka connect (#3660)
rmahindra123 [Tue, 23 Nov 2021 23:48:06 +0000 (15:48 -0800)] 
[HUDI-2325] Add hive sync support to kafka connect (#3660)

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
10 months ago[MINOR] Fix typo,rename 'HooodieAvroDeserializer' to 'HoodieAvroDeserializer' (#4064)
董可伦 [Tue, 23 Nov 2021 11:10:57 +0000 (19:10 +0800)] 
[MINOR] Fix typo,rename 'HooodieAvroDeserializer' to 'HoodieAvroDeserializer' (#4064)

10 months ago[HUDI-2332] Add clustering and compaction in Kafka Connect Sink (#3857)
Y Ethan Guo [Tue, 23 Nov 2021 08:53:28 +0000 (00:53 -0800)] 
[HUDI-2332] Add clustering and compaction in Kafka Connect Sink (#3857)

* [HUDI-2332] Add clustering and compaction in Kafka Connect Sink

* Disable validation check on instant time for compaction and adjust configs

* Add javadocs

* Add clustering and compaction config

* Fix transaction causing missing records in the target table

* Add debugging logs

* Fix kafka offset sync in participant

* Adjust how clustering and compaction are configured in kafka-connect

* Fix clustering strategy

* Remove irrelevant changes from other published PRs

* Update clustering logic and others

* Update README

* Fix test failures

* Fix indentation

* Fix clustering config

* Add JavaCustomColumnsSortPartitioner and make async compaction enabled by default

* Add test for JavaCustomColumnsSortPartitioner

* Add more changes after IDE sync

* Update README with clarification

* Fix clustering logic after rebasing

* Remove unrelated changes

10 months ago[HUDI-2409] Using HBase shaded jars in Hudi presto bundle (#3623)
zhangyue19921010 [Tue, 23 Nov 2021 05:55:12 +0000 (13:55 +0800)] 
[HUDI-2409] Using HBase shaded jars in Hudi presto bundle (#3623)

* using hbase-shaded-jars-in-hudi-presto-hundle

* test

* add hudi-common-bundle

* code review

* code review

* code review

* code review

* test

* test

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
10 months ago[HUDI-2778] Optimize statistics collection related codes and add some docs for z...
xiarixiaoyao [Tue, 23 Nov 2021 05:46:02 +0000 (13:46 +0800)] 
[HUDI-2778] Optimize statistics collection related codes and add some docs for z-order add fix some bugs (#4013)

* [HUDI-2778] Optimize statistics collection related codes and add more docs for z-order.

* add test code for multi-thread parquet footer read

10 months ago[HUDI-2743] Assume path exists and defer fs.exists() in AbstractTableFileSystemView...
Sagar Sumit [Tue, 23 Nov 2021 03:13:10 +0000 (08:43 +0530)] 
[HUDI-2743] Assume path exists and defer fs.exists() in AbstractTableFileSystemView (#4002)

10 months ago[MINOR] Add more configuration to Kafka setup script (#3992)
Y Ethan Guo [Tue, 23 Nov 2021 02:03:38 +0000 (18:03 -0800)] 
[MINOR] Add more configuration to Kafka setup script (#3992)

* [MINOR] Add more configuration to Kafka setup script

* Add option to reuse Kafka topic

* Minor fixes to README

10 months ago[HUDI-1937] Rollback unfinished replace commit to allow updates (#3869)
Sagar Sumit [Tue, 23 Nov 2021 01:59:03 +0000 (07:29 +0530)] 
[HUDI-1937] Rollback unfinished replace commit to allow updates (#3869)

* [HUDI-1937] Rollback unfinished replace commit to allow updates while clustering

* Revert and delete requested replacecommit too

* Rollback pending clustering instants transactionally

* No double locking and add a config to enable rollback

* Update config to be clear about rollback only on conflict

10 months ago[MINOR] Fix typo,'multipe' corrected to 'multiple' (#4068)
Jimmy.Zhou [Tue, 23 Nov 2021 01:20:23 +0000 (09:20 +0800)] 
[MINOR] Fix typo,'multipe' corrected to 'multiple' (#4068)

10 months ago[HUDI-2737] Use earliest instant by default for async compaction and clustering jobs...
Y Ethan Guo [Tue, 23 Nov 2021 01:19:41 +0000 (17:19 -0800)] 
[HUDI-2737] Use earliest instant by default for async compaction and clustering jobs (#3991)

Address review comments

Fix test failures

Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
10 months ago[HUDI-2550] Expand File-Group candidates list for appending for MOR tables (#3986)
Alexey Kudinkin [Tue, 23 Nov 2021 00:19:59 +0000 (16:19 -0800)] 
[HUDI-2550] Expand File-Group candidates list for appending for MOR tables (#3986)

10 months ago[HUDI-2599] Make addFilesToview and fetchLatestBaseFiles public (#4066)
Sagar Sumit [Mon, 22 Nov 2021 17:23:50 +0000 (22:53 +0530)] 
[HUDI-2599] Make addFilesToview and fetchLatestBaseFiles public (#4066)

10 months ago[HUDI-2559] Converting commit timestamp format to millisecs (#4024)
Sivabalan Narayanan [Mon, 22 Nov 2021 16:44:38 +0000 (11:44 -0500)] 
[HUDI-2559] Converting commit timestamp format to millisecs (#4024)

- Adds support for generating commit timestamps with millisecs granularity.
- Older commit timestamps (in secs granularity) will be suffixed with 999 and parsed with millisecs format.

10 months ago[MINOR] Fix instant parsing in HoodieClusteringJob (#4071)
Sagar Sumit [Mon, 22 Nov 2021 13:57:44 +0000 (19:27 +0530)] 
[MINOR] Fix instant parsing in HoodieClusteringJob (#4071)

10 months ago[HUDI-2472] Enabling metadata table for TestHoodieIndex test case (#4045)
Manoj Govindassamy [Mon, 22 Nov 2021 12:21:24 +0000 (04:21 -0800)] 
[HUDI-2472] Enabling metadata table for TestHoodieIndex test case (#4045)

- Enablng the metadata table for testSimpleGlobalIndexTagLocationWhenShouldUpdatePartitionPath.
   This is more of a test issue.

10 months ago[HUDI-2533] New option for hoodieClusteringJob to check, rollback and re-execute...
zhangyue19921010 [Mon, 22 Nov 2021 11:00:33 +0000 (19:00 +0800)] 
[HUDI-2533] New option for hoodieClusteringJob to check, rollback and re-execute the last failed clustering job (#3765)

* coding finished and need to do uts

* add uts

* code review

* code review

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
10 months ago[HUDI-1870] Add more Spark CI build tasks (#4022)
Raymond Xu [Mon, 22 Nov 2021 10:16:45 +0000 (02:16 -0800)] 
[HUDI-1870] Add more Spark CI build tasks  (#4022)

* [HUDI-1870] Add more Spark CI build tasks

- build for spark3.0.x
- build for spark-shade-unbundle-avro
- fix build failures
  - delete unnecessary assertion for spark 3.0.x
  - use AvroConversionUtils#convertAvroSchemaToStructType instead of calling SchemaConverters#toSqlType directly to solve the compilation failures with spark-shade-unbundle-avro (#5)

Co-authored-by: Yann <biyan900116@gmail.com>
10 months ago[HUDI-2799] Fix the classloader of flink write task (#4042)
Danny Chan [Mon, 22 Nov 2021 03:05:05 +0000 (11:05 +0800)] 
[HUDI-2799] Fix the classloader of flink write task (#4042)

10 months ago[MINOR] Fix typos (#4053)
董可伦 [Sun, 21 Nov 2021 08:34:59 +0000 (16:34 +0800)] 
[MINOR] Fix typos (#4053)

10 months ago[HUDI-1932] Update Hive sync timestamp when change detected (#3053)
Nate Radtke [Sun, 21 Nov 2021 06:41:05 +0000 (00:41 -0600)] 
[HUDI-1932] Update Hive sync timestamp when change detected (#3053)

* Update Hive sync timestamp when change detected

Only update the last commit timestamp on the Hive table when the table schema
has changed or a partition is created/updated.

When using AWS Glue Data Catalog as the metastore for Hive this will ensure
that table versions are substantive (including schema and/or partition
changes). Prior to this change when a Hive sync is performed without schema
or partition changes the table in the Glue Data Catalog would have a new
version published with the only change being the timestamp property.

https://issues.apache.org/jira/browse/HUDI-1932

* add conditional sync flag

* fix testSyncWithoutDiffs

* fix HiveSyncConfig

Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>
10 months ago[HUDI-2392] Make flink parquet reader compatible with decimal BINARY encoding (#4057)
Danny Chan [Sun, 21 Nov 2021 05:27:18 +0000 (13:27 +0800)] 
[HUDI-2392] Make flink parquet reader compatible with decimal BINARY encoding (#4057)

10 months ago[HUDI-2804] Add option to skip compaction instants for streaming read (#4051)
Danny Chan [Sun, 21 Nov 2021 04:38:56 +0000 (12:38 +0800)] 
[HUDI-2804] Add option to skip compaction instants for streaming read (#4051)

10 months ago[HUDI-2813] Claim RFC number for RFC for spark datasource V2 Integration (#4059)
leesf [Sun, 21 Nov 2021 02:59:12 +0000 (10:59 +0800)] 
[HUDI-2813] Claim RFC number for RFC for spark datasource V2 Integration (#4059)

10 months ago[MINOR] optimize in constructor of inputbatch class (#4040)
dufeng1010 [Sun, 21 Nov 2021 02:11:01 +0000 (10:11 +0800)] 
[MINOR] optimize in constructor of inputbatch class (#4040)

Co-authored-by: 闫杜峰 <yandufeng@sinochem.com>
10 months ago[MINOR] Claim RFC number for RFC for debezium source for deltastreamer (#4047)
rmahindra123 [Sun, 21 Nov 2021 01:28:48 +0000 (17:28 -0800)] 
[MINOR] Claim RFC number for RFC for debezium source for deltastreamer (#4047)

10 months ago[HUDI-2795] Add mechanism to safely update,delete and recover table properties (...
vinoth chandar [Sat, 20 Nov 2021 16:07:40 +0000 (08:07 -0800)] 
[HUDI-2795] Add mechanism to safely update,delete and recover table properties (#4038)

* [HUDI-2795] Add mechanism to safely update,delete and recover table properties

  - Fail safe mechanism, that lets queries succeed off a backup file
  - Readers who are not upgraded to this version of code will just fail until recovery is done.
  - Added unit tests that exercises all these scenarios.
  - Adding CLI for recovery, updation to table command.
  - [Pending] Add some hash based verfication to ensure any rare partial writes for HDFS

* Fixing upgrade/downgrade infrastructure to use new updation method

10 months ago[HUDI-2742] Added S3 object filter to support multiple S3EventsHoodieIncrSources...
Harsha Teja Kanna [Sat, 20 Nov 2021 09:24:21 +0000 (03:24 -0600)] 
[HUDI-2742] Added S3 object filter to support multiple S3EventsHoodieIncrSources single S3 meta table (#4025)

10 months agoRemove the aws packages from hudi flink bundle jar (#4050)
Ron [Sat, 20 Nov 2021 03:55:12 +0000 (11:55 +0800)] 
Remove the aws packages from hudi flink bundle jar (#4050)

10 months ago[HUDI-2242] Add configuration inference logic for few options (#3359)
wenningd [Sat, 20 Nov 2021 03:38:38 +0000 (19:38 -0800)] 
[HUDI-2242] Add configuration inference logic for few options (#3359)

Co-authored-by: Wenning Ding <wenningd@amazon.com>
10 months ago[HUDI-2796] Metadata table support for Restore action to first commit (#4039)
Manoj Govindassamy [Sat, 20 Nov 2021 01:02:57 +0000 (17:02 -0800)] 
[HUDI-2796] Metadata table support for Restore action to first commit (#4039)

 - Adding support for the metadata table to restore to first commit and
   take proper action for the bootstrap on subequent commits.

10 months ago[HUDI-2472] Enabling metadata table for TestHoodieMergeOnReadTable and TestHoodieComp...
Manoj Govindassamy [Sat, 20 Nov 2021 01:02:21 +0000 (17:02 -0800)] 
[HUDI-2472] Enabling metadata table for TestHoodieMergeOnReadTable and TestHoodieCompactor (#4023)

10 months ago[HUDI-2593] Virtual keys support for metadata table (#3968)
Manoj Govindassamy [Fri, 19 Nov 2021 23:11:29 +0000 (15:11 -0800)] 
[HUDI-2593] Virtual keys support for metadata table (#3968)

- Metadata table today has virtual keys disabled, thereby populating the metafields
  for each record written out and increasing the overall storage space used. Hereby
  adding virtual keys support for metadata table so that metafields are disabled
  for metadata table records.

- Adding a custom KeyGenerator for Metadata table so as to not rely on the
  default Base/SimpleKeyGenerators which currently look for record key
  and partition field set in the table config.

- AbstractHoodieLogRecordReader's version of processing next data block and
  createHoodieRecord() will be a generic version and making the derived class
  HoodieMetadataMergedLogRecordReader take care of the special creation of
  records from explictly passed in partition names.

10 months ago[HUDI-2731] Make clustering work regardless of whether there are base… (#3970)
Sagar Sumit [Fri, 19 Nov 2021 16:09:08 +0000 (21:39 +0530)] 
[HUDI-2731] Make clustering work regardless of whether there are base… (#3970)

10 months ago[HUDI-2798] Fix flink query operation fields (#4041)
Danny Chan [Fri, 19 Nov 2021 15:39:37 +0000 (23:39 +0800)] 
[HUDI-2798] Fix flink query operation fields (#4041)

10 months ago[HUDI-2791] Allows duplicate files for metadata commit (#4033)
Danny Chan [Fri, 19 Nov 2021 06:30:17 +0000 (14:30 +0800)] 
[HUDI-2791] Allows duplicate files for metadata commit (#4033)

10 months ago[HUDI-2641] Avoid deleting all inflight commits heartbeats while rolling back failed...
Udit Mehrotra [Thu, 18 Nov 2021 13:33:50 +0000 (05:33 -0800)] 
[HUDI-2641] Avoid deleting all inflight commits heartbeats while rolling back failed writes (#3956)

10 months ago[HUDI-2362] Add external config file support (#3416)
wenningd [Thu, 18 Nov 2021 09:59:26 +0000 (01:59 -0800)] 
[HUDI-2362] Add external config file support (#3416)

Co-authored-by: Wenning Ding <wenningd@amazon.com>
10 months ago[HUDI-2790] Fix the changelog mode of HoodieTableSource (#4029)
Danny Chan [Thu, 18 Nov 2021 08:40:48 +0000 (16:40 +0800)] 
[HUDI-2790] Fix the changelog mode of HoodieTableSource (#4029)

10 months ago[HUDI-2789] Flink batch upsert for non partitioned table does not work (#4028)
Danny Chan [Thu, 18 Nov 2021 05:59:03 +0000 (13:59 +0800)] 
[HUDI-2789] Flink batch upsert for non partitioned table does not work (#4028)

10 months ago[HUDI-2734] Setting default metadata enable as false for Java (#4003)
Sivabalan Narayanan [Wed, 17 Nov 2021 19:43:00 +0000 (14:43 -0500)] 
[HUDI-2734] Setting default metadata enable as false for Java (#4003)

10 months ago[HUDI-2716] InLineFS support for S3FS logs (#3977)
Manoj Govindassamy [Wed, 17 Nov 2021 18:59:38 +0000 (10:59 -0800)] 
[HUDI-2716] InLineFS support for S3FS logs (#3977)

10 months ago[HUDI-2314] Add support for DynamoDb based lock provider (#3486)
wenningd [Wed, 17 Nov 2021 17:09:31 +0000 (09:09 -0800)] 
[HUDI-2314] Add support for DynamoDb based lock provider (#3486)

- Co-authored-by: Wenning Ding <wenningd@amazon.com>
- Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>

10 months ago[MINOR] Add the Schema for GooseFS to StorageSchemes (#3982)
卢波 [Wed, 17 Nov 2021 14:47:52 +0000 (22:47 +0800)] 
[MINOR] Add the Schema for GooseFS to StorageSchemes (#3982)

Co-authored-by: lubo <bollu@tencent.com>
10 months ago[MINOR] Fix typo,'Hooide' corrected to 'Hoodie' (#4007)
董可伦 [Wed, 17 Nov 2021 08:50:04 +0000 (16:50 +0800)] 
[MINOR] Fix typo,'Hooide' corrected to 'Hoodie' (#4007)

10 months agoCheck --source-avro-schema-path parameter (#3987)
0x574C [Wed, 17 Nov 2021 06:45:43 +0000 (14:45 +0800)] 
Check --source-avro-schema-path  parameter (#3987)

Co-authored-by: 0x3E6 <dragon1996>
10 months ago[HUDI-2151] Part3 Enabling marker based rollback as default rollback strategy (#3950)
Sivabalan Narayanan [Wed, 17 Nov 2021 06:21:28 +0000 (01:21 -0500)] 
[HUDI-2151] Part3 Enabling marker based rollback as default rollback strategy (#3950)

* Enabling timeline server based markers

* Enabling timeline server based markers and marker based rollback

* Removing constraint that timeline server can be enabled only for hdfs

* Fixing tests

10 months ago[HUDI-2753] Ensure list based rollback strategy is used for restore (#3983)
Sivabalan Narayanan [Wed, 17 Nov 2021 04:36:55 +0000 (23:36 -0500)] 
[HUDI-2753] Ensure list based rollback strategy is used for restore (#3983)

10 months ago[MINOR] Fixed checkstyle config to be based off Maven root-dir (requires Maven >...
Alexey Kudinkin [Wed, 17 Nov 2021 02:30:16 +0000 (18:30 -0800)] 
[MINOR] Fixed checkstyle config to be based off Maven root-dir (requires Maven >=3.3.1 to work properly); (#4009)

Updated README

10 months ago[HUDI-2769] Fix StreamerUtil#medianInstantTime for very near instant time (#4005)
Danny Chan [Tue, 16 Nov 2021 05:46:34 +0000 (13:46 +0800)] 
[HUDI-2769] Fix StreamerUtil#medianInstantTime for very near instant time (#4005)

10 months ago[HUDI-2712] Fixing a bug with rollback of partially failed commit which has new parti...
Sivabalan Narayanan [Tue, 16 Nov 2021 03:36:03 +0000 (22:36 -0500)] 
[HUDI-2712] Fixing a bug with rollback of partially failed commit which has new partitions (#3947)

10 months ago[HUDI-2683] Parallelize deleting archived hoodie commits (#3920)
zhangyue19921010 [Mon, 15 Nov 2021 14:36:54 +0000 (22:36 +0800)] 
[HUDI-2683] Parallelize deleting archived hoodie commits (#3920)

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
10 months ago[HUDI-2744] Fix parsing of metadadata table compaction timestamp when metrics are...
Sivabalan Narayanan [Mon, 15 Nov 2021 12:27:35 +0000 (07:27 -0500)] 
[HUDI-2744] Fix parsing of metadadata table compaction timestamp when metrics are enabled (#3976)

10 months ago[MINOR] Fix typo in IntervalTreeBasedGlobalIndexFileFilter (#3993)
dufeng1010 [Mon, 15 Nov 2021 06:39:43 +0000 (14:39 +0800)] 
[MINOR] Fix typo in IntervalTreeBasedGlobalIndexFileFilter (#3993)

Co-authored-by: 闫杜峰 <yandufeng@sinochem.com>
10 months ago[HUDI-2758] remove redundant code in the hoodieRealtimeInputFormatUitls.getRealtimeSp...
xiarixiaoyao [Mon, 15 Nov 2021 03:29:40 +0000 (11:29 +0800)] 
[HUDI-2758] remove redundant code in the hoodieRealtimeInputFormatUitls.getRealtimeSplits (#3994)

10 months ago[HUDI-2589] Claiming RFC-37 for Metadata based bloom index feature. (#3995)
Manoj Govindassamy [Mon, 15 Nov 2021 01:47:41 +0000 (17:47 -0800)] 
[HUDI-2589] Claiming RFC-37 for Metadata based bloom index feature. (#3995)

10 months ago[HUDI-2706] refactor spark-sql to make consistent with DataFrame api (#3936)
Yann Byron [Sun, 14 Nov 2021 23:44:39 +0000 (07:44 +0800)] 
[HUDI-2706] refactor spark-sql to make consistent with DataFrame api (#3936)

10 months ago[HUDI-2756] Fix flink parquet writer decimal type conversion (#3988)
Danny Chan [Sun, 14 Nov 2021 00:51:54 +0000 (08:51 +0800)] 
[HUDI-2756] Fix flink parquet writer decimal type conversion (#3988)

10 months ago[HUDI-2472] Enabling metadata table in TestHoodieIndex and TestMergeOnReadRollbackAct...
Manoj Govindassamy [Sun, 14 Nov 2021 00:37:30 +0000 (16:37 -0800)] 
[HUDI-2472] Enabling metadata table in TestHoodieIndex and TestMergeOnReadRollbackActionExecutor (#3978)

- With rollback after first commit support added to metadata table, these test cases are safe to have metadata table turned on.

10 months ago[HUDI-2697] Minor changes about hbase index config. (#3927)
xiarixiaoyao [Sat, 13 Nov 2021 01:12:33 +0000 (09:12 +0800)] 
[HUDI-2697] Minor changes about hbase index config. (#3927)

10 months ago[HUDI-2741] Fixing instantiating metadata table config in HoodieFileIndex (#3974)
Sivabalan Narayanan [Fri, 12 Nov 2021 22:28:25 +0000 (17:28 -0500)] 
[HUDI-2741] Fixing instantiating metadata table config in HoodieFileIndex (#3974)

10 months ago[HUDI-2718] ExternalSpillableMap payload size re-estimation throws ArithmeticExceptio...
Manoj Govindassamy [Fri, 12 Nov 2021 13:18:40 +0000 (05:18 -0800)] 
[HUDI-2718] ExternalSpillableMap payload size re-estimation throws ArithmeticException (#3955)

- ExternalSpillableMap does the payload/value size estimation on the first put to
  determine when to spill over to disk map. The payload size re-estimation also
  happens after a minimum threshold of puts. This size re-estimation goes my the
  current in-memory map size for calculating average payload size and does attempts
  divide by zero operation when the map is size is empty. Avoiding the
  ArithmeticException during the payload size re-estimate by checking the map size
  upfront.

10 months ago[HUDI-2151] Part1 Setting default parallelism to 200 for some of write configs (...
Sivabalan Narayanan [Fri, 12 Nov 2021 12:29:37 +0000 (07:29 -0500)] 
[HUDI-2151] Part1 Setting default parallelism to 200 for some of write configs (#3948)

10 months ago[HUDI-2746] Do not bootstrap for flink insert overwrite (#3980)
Danny Chan [Fri, 12 Nov 2021 04:17:58 +0000 (12:17 +0800)] 
[HUDI-2746] Do not bootstrap for flink insert overwrite (#3980)

10 months ago[HUDI-2738] Remove the bucketAssignFunction useless context (#3972)
yuzhaojing [Thu, 11 Nov 2021 13:03:01 +0000 (21:03 +0800)] 
[HUDI-2738] Remove the bucketAssignFunction useless context (#3972)

Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
10 months ago[HUDI-2495] Resolve inconsistent key generation for timestamp types by GenericRecord...
Yann Byron [Thu, 11 Nov 2021 03:54:34 +0000 (11:54 +0800)] 
[HUDI-2495] Resolve inconsistent key generation for timestamp types  by GenericRecord and Row (#3944)

10 months ago[HUDI-2634] Improved the metadata table bootstrap for very large tables. (#3873)
Prashant Wason [Thu, 11 Nov 2021 03:37:48 +0000 (19:37 -0800)] 
[HUDI-2634] Improved the metadata table bootstrap for very large tables. (#3873)

* [HUDI-2634] Improved the metadata table bootstrap for very large tables.

Following improvements are implemented:
1. Memory overhead reduction:
  - Existing code caches FileStatus for each file in memory.
  - Created a new class DirectoryInfo which is used to cache a director's file list with parts of the FileStatus (only filename and file len). This reduces the memory requirements.

2. Improved parallelism:
  - Existing code collects all the listing to the Driver and then creates HoodieRecord on the Driver.
  - This takes a long time for large tables (11million HoodieRecords to be created)
  - Created a new function in SparkRDDWriteClient specifically for bootstrap commit. In it, the HoodieRecord creation is parallelized across executors so it completes fast.

3. Fixed setting to limit the number of parallel listings:
  - Existing code had a bug wherein 1500 executors were hardcoded to perform listing. This leads to exception due to limit in the spark's result memory.
  - Corrected the use of the config.

Result:
Dataset has 1299 partitions and 12Million files.
file listing time=1.5mins
HoodieRecord creation time=13seconds
deltacommit duration=2.6mins

Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
10 months ago[HUDI-2685] Support scheduling online compaction plan when there are no commit data...
yuzhaojing [Thu, 11 Nov 2021 02:13:21 +0000 (10:13 +0800)] 
[HUDI-2685] Support scheduling online compaction plan when there are no commit data (#3928)

Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
10 months ago[HUDI-2730] Move EventTimeAvroPayload into hudi-common module (#3959)
yuzhaojing [Wed, 10 Nov 2021 12:22:24 +0000 (20:22 +0800)] 
[HUDI-2730] Move EventTimeAvroPayload into hudi-common module (#3959)

Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
10 months ago[HUDI-2442] Change default values for certin clustering configs (#3875)
Sagar Sumit [Wed, 10 Nov 2021 08:53:24 +0000 (14:23 +0530)] 
[HUDI-2442] Change default values for certin clustering configs (#3875)

10 months ago[HUDI-2086]redo the logical of mor_incremental_view for hive (#3203)
xiarixiaoyao [Wed, 10 Nov 2021 07:41:07 +0000 (15:41 +0800)] 
[HUDI-2086]redo the logical of mor_incremental_view for hive (#3203)

10 months ago[HUDI-2297] Estimate available memory size for spillable map accurately. (#3455)
Shawy Geng [Wed, 10 Nov 2021 06:05:12 +0000 (14:05 +0800)] 
[HUDI-2297] Estimate available memory size for spillable map accurately. (#3455)

10 months ago[HUDI-1877] Support records staying in same fileId after clustering (#3833)
Sagar Sumit [Wed, 10 Nov 2021 04:17:50 +0000 (09:47 +0530)] 
[HUDI-1877] Support records staying in same fileId after clustering (#3833)

* [HUDI-1877] Support records staying in same fileId after clustering

Add plan strategy

* Ensure same filegroup id and refactor based on comments

10 months ago[HUDI-2579] Make deltastreamer checkpoint state merging more explicit (#3820)
davehagman [Tue, 9 Nov 2021 22:37:59 +0000 (17:37 -0500)] 
[HUDI-2579] Make deltastreamer checkpoint state merging more explicit (#3820)

Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>