hudi.git
8 months ago[MINOR] Fix some code style issues based on check-style plugin (#4532)
YueZhang [Sun, 9 Jan 2022 09:14:56 +0000 (17:14 +0800)] 
[MINOR] Fix some code style issues based on check-style plugin (#4532)

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
8 months ago[HUDI-3125] spark-sql write timestamp directly (#4471)
Yann Byron [Sun, 9 Jan 2022 07:43:25 +0000 (15:43 +0800)] 
[HUDI-3125] spark-sql write timestamp directly (#4471)

8 months ago[HUDI-3104] Kafka-connect support of hadoop config environments and properties (...
Thinking Chen [Sun, 9 Jan 2022 07:10:17 +0000 (15:10 +0800)] 
[HUDI-3104] Kafka-connect support of hadoop config environments and properties (#4451)

8 months ago[HUDI-3178] Fixing metadata table compaction so as to not include uncommitted data...
Sivabalan Narayanan [Sat, 8 Jan 2022 15:34:47 +0000 (10:34 -0500)] 
[HUDI-3178] Fixing metadata table compaction so as to not include uncommitted data (#4530)

- There is a chance that the actual write eventually failed in data table but commit was successful in Metadata table, and if compaction was triggered in MDT, compaction could have included the uncommitted data. But once compacted, it may never be ignored while reading from metadata table. So, this patch fixes the bug. Metadata table compaction is triggered before applying the commit to metadata table to circumvent this issue.

8 months ago[HUDI-3139] Shade htrace and parquet-avro in presto bundle (#4495)
Sagar Sumit [Sat, 8 Jan 2022 15:29:36 +0000 (20:59 +0530)] 
[HUDI-3139] Shade htrace and parquet-avro in presto bundle (#4495)

Filter out unnecessary classes

8 months ago[HUDI-2909] Handle logical type in TimestampBasedKeyGenerator (#4203)
Sagar Sumit [Sat, 8 Jan 2022 15:22:44 +0000 (20:52 +0530)] 
[HUDI-2909] Handle logical type in TimestampBasedKeyGenerator (#4203)

* [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator

Timestampbased key generator was returning diff values for row writer and non row writer path. this patch fixes it and is guarded by a config flag (`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled`)

8 months ago[HUDI-3195] optimize spark3 pom and modify build command (#4538)
Yann Byron [Sat, 8 Jan 2022 07:21:39 +0000 (15:21 +0800)] 
[HUDI-3195] optimize spark3 pom and modify build command (#4538)

8 months ago[HUDI-3192] Spark metastore schema evolution broken (#4533)
董可伦 [Sat, 8 Jan 2022 02:48:37 +0000 (10:48 +0800)] 
[HUDI-3192] Spark metastore schema evolution broken (#4533)

8 months ago[HUDI-3185] HoodieConfig#getBoolean should return false when default not set (#4536)
Sagar Sumit [Fri, 7 Jan 2022 21:20:11 +0000 (02:50 +0530)] 
[HUDI-3185] HoodieConfig#getBoolean should return false when default not set (#4536)

Remove unnecessary config

8 months ago[HUDI-2947] Fixing checkpoint fetch in detlastreamer (#4485)
Sivabalan Narayanan [Fri, 7 Jan 2022 16:38:58 +0000 (11:38 -0500)] 
[HUDI-2947] Fixing checkpoint fetch in detlastreamer (#4485)

* Fixing checkpoint fetch in detlastreamer

* Addressing comments

8 months ago[MINOR] fix typos in DDLExecutor (#4534)
董可伦 [Fri, 7 Jan 2022 12:59:55 +0000 (20:59 +0800)] 
[MINOR] fix typos in DDLExecutor (#4534)

8 months ago[HUDI-3188] Update quick start guide for Kafka Connect Sink for Hudi (#4527)
Y Ethan Guo [Fri, 7 Jan 2022 12:56:08 +0000 (04:56 -0800)] 
[HUDI-3188] Update quick start guide for Kafka Connect Sink for Hudi (#4527)

8 months ago[HUDI-3100] Add config for hive conditional sync (#4440)
Raymond Xu [Fri, 7 Jan 2022 07:26:35 +0000 (23:26 -0800)] 
[HUDI-3100] Add config for hive conditional sync (#4440)

8 months ago[HUDI-3183] Wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter...
YueZhang [Fri, 7 Jan 2022 02:16:29 +0000 (10:16 +0800)] 
[HUDI-3183] Wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter (#4521)

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
8 months ago[HUDI-3118] Add default HUDI_DIR in setupKafka.sh (#4460)
Thinking Chen [Thu, 6 Jan 2022 23:46:51 +0000 (07:46 +0800)] 
[HUDI-3118] Add default HUDI_DIR in setupKafka.sh (#4460)

8 months ago[MINOR] Remove unused methods in HoodieColumnProjectionUtils (#4408)
xuzifu666 [Thu, 6 Jan 2022 23:36:13 +0000 (07:36 +0800)] 
[MINOR] Remove unused methods in HoodieColumnProjectionUtils (#4408)

8 months ago[HUDI-3165] Enabling InProcessLockProvider for all multi-writer tests instead of...
Sivabalan Narayanan [Thu, 6 Jan 2022 18:04:10 +0000 (13:04 -0500)] 
[HUDI-3165] Enabling InProcessLockProvider for all multi-writer tests instead of FileSystemBasedLockProviderTestClass (#4427)

8 months ago[HUDI-52] Enabling savepoint and restore for MOR table (#4507)
Sivabalan Narayanan [Thu, 6 Jan 2022 15:56:08 +0000 (10:56 -0500)] 
[HUDI-52] Enabling savepoint and restore for MOR table (#4507)

* Enabling restore for MOR table

* Fixing savepoint for compaction commits in MOR

8 months ago[HUDI-44] Adding support to preserve commit metadata for compaction (#4428)
Sivabalan Narayanan [Thu, 6 Jan 2022 14:57:37 +0000 (09:57 -0500)] 
[HUDI-44] Adding support to preserve commit metadata for compaction (#4428)

8 months agoUpdate HiveIncrementalPuller to configure filesystem (#4431)
hehexiaoduantui [Thu, 6 Jan 2022 07:49:30 +0000 (15:49 +0800)] 
Update HiveIncrementalPuller to configure filesystem (#4431)

* Update HiveIncrementalPuller.java

fix get FileSystem bug

* Update HiveIncrementalPuller.java

fix error

* Update HiveIncrementalPuller.java

fie error

8 months ago[HUDI-3132] Minor fixes for HoodieCatalog
fengli [Fri, 31 Dec 2021 15:25:04 +0000 (23:25 +0800)] 
[HUDI-3132] Minor fixes for HoodieCatalog

close apache/hudi#4486

8 months ago[HUDI-3168] Fixing null schema with empty commit in incremental relation (#4513)
Vinish Reddy [Wed, 5 Jan 2022 16:43:10 +0000 (22:13 +0530)] 
[HUDI-3168] Fixing null schema with empty commit in incremental relation (#4513)

8 months ago[HUDI-3170] Do not preserve filename when preserveCommitMetadata enabled (#4512)
Sagar Sumit [Wed, 5 Jan 2022 13:09:58 +0000 (18:39 +0530)] 
[HUDI-3170] Do not preserve filename when preserveCommitMetadata enabled (#4512)

8 months ago[HUDI-3171] Sync empty table to hive metastore (#4511)
Danny Chan [Wed, 5 Jan 2022 08:41:33 +0000 (16:41 +0800)] 
[HUDI-3171] Sync empty table to hive metastore (#4511)

8 months ago[HUDI-2966] Closing LogRecordScanner in compactor (#4478)
Sivabalan Narayanan [Wed, 5 Jan 2022 02:57:18 +0000 (21:57 -0500)] 
[HUDI-2966] Closing LogRecordScanner in compactor (#4478)

* Closing LogRecordScanner in compactor

* Addressing comments

8 months ago[HUDI-3147] Add endpoint_url to dynamodb lock provider (#4500)
Nicolas Paris [Tue, 4 Jan 2022 21:42:28 +0000 (22:42 +0100)] 
[HUDI-3147] Add endpoint_url to dynamodb lock provider (#4500)

Co-authored-by: Nicolas Paris <nicolas.paris@adevinta.com>
8 months ago[HUDI-3141] Metadata merged log record reader - avoiding NullPointerException when...
Manoj Govindassamy [Tue, 4 Jan 2022 21:41:33 +0000 (13:41 -0800)] 
[HUDI-3141] Metadata merged log record reader - avoiding NullPointerException when records by keys (#4505)

- HoodieMetadataMergedLogRecordReader#getRecordsByKeys() and its parent class methods
   are not thread safe. When multiple queries come in for gettting log records
   by keys, they all operate on the same log record reader instance provided by
   HoodieBackedTableMetadata#openReadersIfNeeded() and they trip over each other
   as they clear/put/get the same class memeber records.

 - The fix is to streamline the mutatation to class member records. Making
   HoodieMetadataMergedLogRecordReader#getRecordsByKeys() a synchronized method
to avoid concurrent log records readers getting into NPE.

8 months ago[HUDI-2774] Handle duplicate instants when fetching pending clustering plans (#4118)
Sagar Sumit [Tue, 4 Jan 2022 21:32:05 +0000 (03:02 +0530)] 
[HUDI-2774] Handle duplicate instants when fetching pending clustering plans (#4118)

8 months agoAdding tests to validate different key generators (#4473)
Sivabalan Narayanan [Tue, 4 Jan 2022 05:18:04 +0000 (00:18 -0500)] 
Adding tests to validate different key generators (#4473)

8 months ago[HUDI-3140] Fix bulk_insert failure on Spark 3.2.0 (#4498)
leesf [Tue, 4 Jan 2022 01:59:59 +0000 (09:59 +0800)] 
[HUDI-3140] Fix bulk_insert failure on Spark 3.2.0 (#4498)

8 months ago[HUDI-2558] Fixing Clustering w/ sort columns with null values fails (#4404)
harshal [Mon, 3 Jan 2022 06:49:43 +0000 (12:19 +0530)] 
[HUDI-2558] Fixing Clustering w/ sort columns with null values fails (#4404)

8 months ago[MINOR] Update README.md (#4492)
Raymond Xu [Mon, 3 Jan 2022 04:34:37 +0000 (20:34 -0800)] 
[MINOR] Update README.md (#4492)

Update Spark 3 build instructions

8 months ago[HUDI-3138] Fix broken UT test for TestHiveSyncTool.testDropPartitions (#4493)
YueZhang [Mon, 3 Jan 2022 03:43:30 +0000 (11:43 +0800)] 
[HUDI-3138] Fix broken UT test for TestHiveSyncTool.testDropPartitions (#4493)

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
8 months ago[HUDI-3131] fix ctas error in spark3.1.1 (#4476)
Yann Byron [Sun, 2 Jan 2022 11:06:55 +0000 (19:06 +0800)] 
[HUDI-3131] fix ctas error in spark3.1.1 (#4476)

8 months ago[HUDI-3136] Fix merge/insert/show partitions error on Spark3.2 (#4490)
Yann Byron [Sun, 2 Jan 2022 10:42:10 +0000 (18:42 +0800)] 
[HUDI-3136] Fix merge/insert/show partitions error on Spark3.2 (#4490)

8 months ago[HUDI-3134] Fix insert error after adding columns on Spark 3.2.0 (#4488)
leesf [Sun, 2 Jan 2022 01:38:14 +0000 (09:38 +0800)] 
[HUDI-3134] Fix insert error after adding columns on Spark 3.2.0 (#4488)

8 months ago[HUDI-3040] Fix HoodieSparkBootstrapExample error info for usage (#4341)
Aimiyoo [Sat, 1 Jan 2022 07:38:38 +0000 (15:38 +0800)] 
[HUDI-3040] Fix HoodieSparkBootstrapExample error info for usage (#4341)

8 months ago[HUDI-3107]Fix HiveSyncTool drop partitions using JDBC or hivesql or hms (#4453) 4482/head
YueZhang [Fri, 31 Dec 2021 07:56:33 +0000 (15:56 +0800)] 
[HUDI-3107]Fix HiveSyncTool drop partitions using JDBC or hivesql or hms (#4453)

* constructDropPartitions when drop partitions using jdbc

* done

* done

* code style

* code review

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
8 months ago[HUDI-3095] abstract partition filter logic to enable code reuse (#4454)
Yuwei XIAO [Fri, 31 Dec 2021 05:37:52 +0000 (13:37 +0800)] 
[HUDI-3095] abstract partition filter logic to enable code reuse (#4454)

* [HUDI-3095] abstract partition filter logic to enable code reuse

* [HUDI-3095] address reviews

8 months ago[HUDI-3120] Cache compactionPlan in buffer (#4463)
yuzhaojing [Fri, 31 Dec 2021 05:12:32 +0000 (13:12 +0800)] 
[HUDI-3120] Cache compactionPlan in buffer (#4463)

Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
8 months ago[HUDI-1951] Add bucket hash index, compatible with the hive bucket (#3173)
Shawy Geng [Thu, 30 Dec 2021 20:38:26 +0000 (04:38 +0800)] 
[HUDI-1951] Add bucket hash index, compatible with the hive bucket (#3173)

* [HUDI-2154] Add index key field to HoodieKey

* [HUDI-2157] Add the bucket index and its read/write implemention of Spark engine.
* revert HUDI-2154 add index key field to HoodieKey
* fix all comments and introduce a new tricky way to get index key at runtime
support double insert for bucket index
* revert spark read optimizer based on bucket index
* add the storage layout
* index tag, hash function and add ut
* fix ut
* address partial comments
* Code review feedback
* add layout config and docs
* fix ut
* rename hoodie.layout and rebase master

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
8 months ago[HUDI-3124] Bootstrap when timeline have completed instant (#4467)
yuzhaojing [Thu, 30 Dec 2021 03:54:34 +0000 (11:54 +0800)] 
[HUDI-3124] Bootstrap when timeline have completed instant (#4467)

Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
8 months ago[HUDI-2675] Fix the exception 'Not an Avro data file' when archive and clean (#4016)
董可伦 [Thu, 30 Dec 2021 03:53:17 +0000 (11:53 +0800)] 
[HUDI-2675] Fix the exception 'Not an Avro data file' when archive and clean (#4016)

8 months ago[HUDI-3083] Support component data types for flink bulk_insert (#4470)
Ron [Thu, 30 Dec 2021 03:15:54 +0000 (11:15 +0800)] 
[HUDI-3083] Support component data types for flink bulk_insert (#4470)

* [HUDI-3083] Support component data types for flink bulk_insert

* add nested row type test

8 months agoRevert "[HUDI-3043] Revert async cleaner leak commit to unblock CI failure (#4343...
Sivabalan Narayanan [Thu, 30 Dec 2021 02:45:09 +0000 (21:45 -0500)] 
Revert "[HUDI-3043] Revert async cleaner leak commit to unblock CI failure (#4343)" (#4465)

This reverts commit 7e7ad1558c0dcc06e059f631e43e44dc04100aa4.

8 months ago[HUDI-3108] Fix Purge Drop MOR Table Cause error (#4455)
ForwardXu [Wed, 29 Dec 2021 12:23:23 +0000 (20:23 +0800)] 
[HUDI-3108] Fix Purge Drop MOR Table Cause error (#4455)

8 months ago[MINOR] HoodieInstantTimeGenerator improve method used (#4462)
xuzifu666 [Wed, 29 Dec 2021 10:43:16 +0000 (18:43 +0800)] 
[MINOR] HoodieInstantTimeGenerator improve method used (#4462)

8 months ago[HUDI-2983] Remove Log4j2 transitive dependencies (#4281)
Udit Mehrotra [Tue, 28 Dec 2021 15:15:05 +0000 (07:15 -0800)] 
[HUDI-2983] Remove Log4j2 transitive dependencies (#4281)

8 months agoFixing dynamoDbLockConfig required prop check (#4422)
Sivabalan Narayanan [Tue, 28 Dec 2021 10:26:30 +0000 (05:26 -0500)] 
Fixing dynamoDbLockConfig required prop check (#4422)

8 months ago[HUDI-2811] Support Spark 3.2 (#4270)
Yann Byron [Tue, 28 Dec 2021 08:12:44 +0000 (16:12 +0800)] 
[HUDI-2811] Support Spark 3.2 (#4270)

8 months ago[HUDI-3106] Fix HiveSyncTool not sync schema (#4452)
ForwardXu [Tue, 28 Dec 2021 06:11:14 +0000 (14:11 +0800)] 
[HUDI-3106] Fix HiveSyncTool not sync schema (#4452)

8 months ago[HUDI-3093] fix spark-sql query table that write with TimestampBasedKeyGenerator...
Yann Byron [Tue, 28 Dec 2021 05:39:52 +0000 (13:39 +0800)] 
[HUDI-3093] fix spark-sql query table that write with TimestampBasedKeyGenerator (#4416)

8 months ago[HUDI-2374] Fixing AvroDFSSource does not use the overridden schema to deserialize...
harshal [Tue, 28 Dec 2021 04:01:21 +0000 (09:31 +0530)] 
[HUDI-2374] Fixing AvroDFSSource does not use the overridden schema to deserialize Avro binaries (#4353)

8 months ago[HUDI-3099] Purge drop partition for spark sql (#4436)
ForwardXu [Tue, 28 Dec 2021 01:38:26 +0000 (09:38 +0800)] 
[HUDI-3099] Purge drop partition for spark sql (#4436)

9 months ago[HUDI-3102] Do not store rollback plan in inflight instant (#4445)
Danny Chan [Sat, 25 Dec 2021 10:10:43 +0000 (18:10 +0800)] 
[HUDI-3102] Do not store rollback plan in inflight instant (#4445)

9 months ago[HUDI-3101] Excluding compaction instants from pending rollback info (#4443)
Danny Chan [Sat, 25 Dec 2021 06:10:45 +0000 (14:10 +0800)] 
[HUDI-3101] Excluding compaction instants from pending rollback info (#4443)

9 months ago[MINOR] Remove unused method in HoodieActiveTimeline (#4435)
xuzifu666 [Fri, 24 Dec 2021 14:29:34 +0000 (22:29 +0800)] 
[MINOR] Remove unused method in HoodieActiveTimeline (#4435)

9 months ago[HUDI-3014] Add table option to set utc timezone (#4306)
xuzifu666 [Thu, 23 Dec 2021 08:27:45 +0000 (16:27 +0800)] 
[HUDI-3014] Add table option to set utc timezone (#4306)

9 months ago[MINOR] Fix DedupeSparkJob typo (#4418)
Aimiyoo [Wed, 22 Dec 2021 19:51:26 +0000 (03:51 +0800)] 
[MINOR] Fix DedupeSparkJob typo (#4418)

9 months ago[HUDI-3060] drop table for spark sql (#4364)
ForwardXu [Wed, 22 Dec 2021 11:17:43 +0000 (19:17 +0800)] 
[HUDI-3060] drop table for spark sql (#4364)

9 months ago[HUDI-3011] Adding ability to read entire data with HoodieIncrSource with empty check...
Sivabalan Narayanan [Wed, 22 Dec 2021 10:13:06 +0000 (02:13 -0800)] 
[HUDI-3011] Adding ability to read entire data with HoodieIncrSource with empty checkpoint (#4334)

* Adding ability to read entire data with HoodieIncrSource with empty checkpoint

* Addressing comments

9 months agoMerge pull request #4308 from harsh1231/HUDI-3008
xiarixiaoyao [Wed, 22 Dec 2021 08:46:57 +0000 (16:46 +0800)] 
Merge pull request #4308 from harsh1231/HUDI-3008

[HUDI-3008] Fixing HoodieFileIndex partition column parsing for nested fields

9 months ago[HUDI-2547] Schedule Flink compaction in service (#4254)
yuzhaojing [Wed, 22 Dec 2021 07:08:47 +0000 (15:08 +0800)] 
[HUDI-2547] Schedule Flink compaction in service (#4254)

Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
9 months ago[HUDI-3032] Do not clean the log files right after compaction for metadata table...
Danny Chan [Wed, 22 Dec 2021 03:10:27 +0000 (11:10 +0800)] 
[HUDI-3032] Do not clean the log files right after compaction for metadata table (#4336)

9 months ago[HUDI-3027] Update hudi-examples README.md (#4330)
Aimiyoo [Tue, 21 Dec 2021 21:36:03 +0000 (05:36 +0800)] 
[HUDI-3027] Update hudi-examples README.md (#4330)

9 months ago[HUDI-3008] Fixing HoodieFileIndex partition column parsing for nested fields 4308/head
harshal patil [Tue, 14 Dec 2021 11:58:18 +0000 (17:28 +0530)] 
[HUDI-3008] Fixing HoodieFileIndex partition column parsing for nested fields

9 months ago[HUDI-2970] Add test for archiving replace commit (#4345)
Raymond Xu [Tue, 21 Dec 2021 05:01:59 +0000 (21:01 -0800)] 
[HUDI-2970] Add test for archiving replace commit (#4345)

9 months ago[HUDI-3070] Add rerunFailingTestsCount for flakly testes (#4398)
zhangyue19921010 [Tue, 21 Dec 2021 03:59:50 +0000 (11:59 +0800)] 
[HUDI-3070] Add rerunFailingTestsCount for flakly testes (#4398)

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
9 months ago[MINOR] Increasing CI timeout to 90 mins (#4407)
Sivabalan Narayanan [Tue, 21 Dec 2021 01:27:22 +0000 (17:27 -0800)] 
[MINOR] Increasing CI timeout to 90 mins (#4407)

9 months ago[MINOR] Remove unused method in HoodieActiveTimeline (#4401)
xuzifu666 [Mon, 20 Dec 2021 14:19:37 +0000 (22:19 +0800)] 
[MINOR] Remove unused method in HoodieActiveTimeline (#4401)

9 months agoremove unused import (#4349)
xuzifu666 [Mon, 20 Dec 2021 08:32:41 +0000 (16:32 +0800)] 
remove unused import (#4349)

9 months ago[HUDI-3064][HUDI-3054] FileSystemBasedLockProviderTestClass tryLock fix and TestHoodi...
Manoj Govindassamy [Sun, 19 Dec 2021 18:31:02 +0000 (10:31 -0800)] 
[HUDI-3064][HUDI-3054] FileSystemBasedLockProviderTestClass tryLock fix and TestHoodieClientMultiWriter test fixes (#4384)

 - Made FileSystemBasedLockProviderTestClass thread safe and fixed the
   tryLock retry logic.

 - Made TestHoodieClientMultiWriter. testHoodieClientBasicMultiWriter
   deterministic in verifying the HoodieWriteConflictException.

9 months ago[HUDI-2970] Adding tests for archival of replace commit actions (#4268)
Sivabalan Narayanan [Sun, 19 Dec 2021 07:59:39 +0000 (23:59 -0800)] 
[HUDI-2970] Adding tests for archival of replace commit actions (#4268)

9 months ago[minor] fix NetworkUtils#getHostname (#4355)
Danny Chan [Sun, 19 Dec 2021 02:09:48 +0000 (10:09 +0800)] 
[minor] fix NetworkUtils#getHostname (#4355)

9 months ago[HUDI-3052] Fix flaky testJsonKafkaSourceResetStrategy (#4381)
Raymond Xu [Sun, 19 Dec 2021 01:58:51 +0000 (17:58 -0800)] 
[HUDI-3052] Fix flaky testJsonKafkaSourceResetStrategy (#4381)

9 months ago[MINOR] Azure CI IT tasks clean up (#4337)
Raymond Xu [Sun, 19 Dec 2021 01:00:56 +0000 (17:00 -0800)] 
[MINOR] Azure CI IT tasks clean up (#4337)

9 months ago[HUDI-3054] Fixing default lock configs for FileSystemBasedLock and fixing a flaky...
Sivabalan Narayanan [Sat, 18 Dec 2021 21:15:48 +0000 (13:15 -0800)] 
[HUDI-3054] Fixing default lock configs for FileSystemBasedLock and fixing a flaky test (#4374)

9 months ago[HUDI-3064] Fixing a bug in TransactionManager and FileSystemTestLock (#4372)
Sivabalan Narayanan [Sat, 18 Dec 2021 16:52:11 +0000 (08:52 -0800)] 
[HUDI-3064] Fixing a bug in TransactionManager and FileSystemTestLock (#4372)

9 months ago[HUDI-3029] Transaction manager: avoid deadlock when doing begin and end transaction...
Manoj Govindassamy [Sat, 18 Dec 2021 16:43:10 +0000 (08:43 -0800)] 
[HUDI-3029]  Transaction manager: avoid deadlock when doing begin and end transactions (#4373)

9 months ago[HUDI-3029] Transaction manager: avoid deadlock when doing begin and end transaction...
Manoj Govindassamy [Sat, 18 Dec 2021 14:43:17 +0000 (06:43 -0800)] 
[HUDI-3029]  Transaction manager: avoid deadlock when doing begin and end transactions (#4363)

* [HUDI-3029] Transaction manager: avoid deadlock when doing begin and end transactions

 - Transaction manager has begin and end transactions as synchronized methods.
   Based on the lock provider implementaion, this can lead to deadlock
   situation when the underlying lock() calls are blocking or with a long timeout.

 - Fixing transaction manager begin and end transactions to not get to deadlock
   and to not assume anything on the lock provider implementation.

9 months ago[HUDI-3043] De-coupling multi writer tests (#4362)
Sivabalan Narayanan [Sat, 18 Dec 2021 02:37:45 +0000 (18:37 -0800)] 
[HUDI-3043] De-coupling multi writer tests (#4362)

9 months ago[HUDI-2962] InProcess lock provider to guard single writer process with async table...
Manoj Govindassamy [Sat, 18 Dec 2021 01:18:46 +0000 (17:18 -0800)] 
[HUDI-2962] InProcess lock provider to guard single writer process with async table operations (#4259)

 - Adding Local JVM process based lock provider implementation

 - This local lock provider can be used by a single writer process with async
   table operations to guard the metadata tabl against concurrent updates.

9 months ago[HUDI-3043] Adding some test fixes to continuous mode multi writer tests (#4356)
Sivabalan Narayanan [Fri, 17 Dec 2021 20:45:05 +0000 (15:45 -0500)] 
[HUDI-3043] Adding some test fixes to continuous mode multi writer tests (#4356)

9 months ago[HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkins...
xiarixiaoyao [Fri, 17 Dec 2021 13:58:02 +0000 (21:58 +0800)] 
[HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType (#4253)

9 months ago[HUDI-3046] Claim RFC number for RFC for Compaction / Clustering Service (#4347)
yuzhaojing [Fri, 17 Dec 2021 06:52:45 +0000 (14:52 +0800)] 
[HUDI-3046] Claim RFC number for RFC for Compaction / Clustering Service (#4347)

Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
9 months ago[HUDI-3037] Add back remote view storage config for flink (#4338)
Danny Chan [Fri, 17 Dec 2021 05:57:53 +0000 (13:57 +0800)] 
[HUDI-3037] Add back remote view storage config for flink (#4338)

9 months ago[HUDI-3043] Revert async cleaner leak commit to unblock CI failure (#4343)
Sivabalan Narayanan [Fri, 17 Dec 2021 02:51:28 +0000 (21:51 -0500)] 
[HUDI-3043] Revert async cleaner leak commit to unblock CI failure (#4343)

* Revert "[HUDI-2959] Fix the thread leak of cleaning service (#4252)"
Reverting to unblock CI failure for now. will revisit this with the right fix

9 months ago[HUDI-3001] Clean up the marker directory when finish bootstrap operation. (#4298)
xiarixiaoyao [Thu, 16 Dec 2021 20:36:01 +0000 (04:36 +0800)] 
[HUDI-3001] Clean up the marker directory when finish bootstrap operation. (#4298)

9 months ago[Minor] Catch and ignore all the exceptions in quietDeleteMarkerDir (#4301)
zhangyue19921010 [Thu, 16 Dec 2021 19:15:08 +0000 (03:15 +0800)] 
[Minor] Catch and ignore all the exceptions in quietDeleteMarkerDir (#4301)

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
9 months ago[HUDI-3015] Implement #reset and #sync for metadata filesystem view (#4307)
Danny Chan [Thu, 16 Dec 2021 07:26:16 +0000 (15:26 +0800)] 
[HUDI-3015] Implement #reset and #sync for metadata filesystem view (#4307)

9 months ago[HUDI-2998] claiming rfc number for consistent hashing index (#4303)
Yuwei XIAO [Wed, 15 Dec 2021 23:47:22 +0000 (07:47 +0800)] 
[HUDI-2998] claiming rfc number for consistent hashing index (#4303)

Co-authored-by: xiaoyuwei <xiaoyuwei.yw@alibaba-inc.com>
9 months ago[HUDI-3028] Use blob storage to speed up CI downloads (#4331)
Raymond Xu [Wed, 15 Dec 2021 23:33:33 +0000 (15:33 -0800)] 
[HUDI-3028] Use blob storage to speed up CI downloads (#4331)

Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
9 months ago[HUDI-3025] Add additional wait time for namenode availability during IT tests initia...
Y Ethan Guo [Wed, 15 Dec 2021 18:44:42 +0000 (10:44 -0800)] 
[HUDI-3025] Add additional wait time for namenode availability during IT tests initiatialization  (#4328)

- Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>

9 months ago[HUDI-3024] Add explicit write handler for flink (#4329)
WangMinChao [Wed, 15 Dec 2021 12:16:48 +0000 (20:16 +0800)] 
[HUDI-3024] Add explicit write handler for flink (#4329)

Co-authored-by: wangminchao <wangminchao@asinking.com>
9 months ago[HUDI-3022] Fix NPE for isDropPartition method (#4319)
ForwardXu [Wed, 15 Dec 2021 11:38:02 +0000 (19:38 +0800)] 
[HUDI-3022] Fix NPE for isDropPartition method (#4319)

* [HUDI-3022] Fix NPE for isDropPartition method

9 months ago[HUDI-2995] Enabling metadata table by default (#4295)
Manoj Govindassamy [Tue, 14 Dec 2021 21:11:19 +0000 (13:11 -0800)] 
[HUDI-2995] Enabling metadata table by default (#4295)

- Enabling metadata table by default

9 months ago[HUDI-2997] Skip the corrupt meta file for pending rollback action (#4296)
Danny Chan [Tue, 14 Dec 2021 06:08:13 +0000 (14:08 +0800)] 
[HUDI-2997] Skip the corrupt meta file for pending rollback action (#4296)

9 months ago[HUDI-2996] Flink streaming reader 'skip_compaction' option does not work (#4304)
Fugle666 [Tue, 14 Dec 2021 03:31:36 +0000 (11:31 +0800)] 
[HUDI-2996] Flink streaming reader 'skip_compaction' option does not work (#4304)

close apache/hudi#4304

9 months ago[HUDI-2994] Add judgement to existed partitionPath in the catch code block for HU...
WangMinChao [Mon, 13 Dec 2021 12:41:03 +0000 (20:41 +0800)] 
[HUDI-2994] Add judgement to existed partitionPath in the catch code block for HU… (#4294)

* [HUDI-2994] Add judgement to existed partition path in the catch code block for HUDI-2743

Co-authored-by: wangminchao <wangminchao@asinking.com>
9 months ago[HUDI-2990] Sync to HMS when deleting partitions (#4291)
ForwardXu [Mon, 13 Dec 2021 12:40:06 +0000 (20:40 +0800)] 
[HUDI-2990] Sync to HMS when deleting partitions (#4291)