hudi.git
6 months ago[MINOR] Update release version to reflect published version 0.10.0 release-0.10.0 release-0.10.0
yuzhao.cyz [Wed, 8 Dec 2021 02:31:31 +0000 (10:31 +0800)] 
[MINOR] Update release version to reflect published version 0.10.0

6 months agoBumping release candidate number 3
yuzhao.cyz [Wed, 8 Dec 2021 02:29:03 +0000 (10:29 +0800)] 
Bumping release candidate number 3

6 months ago[HUDI-2951] Disable remote view storage config for flink (#4237)
Danny Chan [Tue, 7 Dec 2021 10:04:15 +0000 (18:04 +0800)] 
[HUDI-2951] Disable remote view storage config for flink (#4237)

(cherry picked from commit e8473b9a2b5bf0ad9370377899f6a7ea4d1ceba1)

6 months ago[HUDI-2935] Remove special casing of clustering in deltastreamer checkpoint retrival...
vinoth chandar [Sat, 4 Dec 2021 09:16:11 +0000 (01:16 -0800)] 
[HUDI-2935] Remove special casing of clustering in deltastreamer checkpoint retrival (#4216)

- We now seek backwards to find the checkpoint
 - No need to return empty anymore

(cherry picked from commit 36b69d80332468439d9e1674b7d1845198379152)

6 months ago[HUDI-2934] Optimize RequestHandler code style
fengli [Sat, 4 Dec 2021 05:56:25 +0000 (13:56 +0800)] 
[HUDI-2934] Optimize RequestHandler code style

close apache/hudi#4215

(cherry picked from commit 568181a3e7c1d43a4ee8573d3746600be4f94379)

6 months ago[HUDI-2923] Fixing metadata table reader when metadata compaction is inflight (#4206)
Sivabalan Narayanan [Sat, 4 Dec 2021 05:44:50 +0000 (00:44 -0500)] 
[HUDI-2923] Fixing metadata table reader when metadata compaction is inflight (#4206)

* [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight

* Fixing retry of pending compaction in metadata table and enhancing tests

(cherry picked from commit 1d4fb827e73b2ae510f0e2c6510a448c0b5bd5b3)

6 months ago[HUDI-2890] Kafka Connect: Fix failed writes and avoid table service concurrent opera...
rmahindra123 [Sat, 4 Dec 2021 05:30:32 +0000 (21:30 -0800)] 
[HUDI-2890] Kafka Connect: Fix failed writes and avoid table service concurrent operations (#4211)

* Fix kafka connect readme

* Fix handling of errors in write records for kafka connect

* By default, ensure we skip error records and keep the pipeline alive

* Fix indentation

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
(cherry picked from commit 94f45e928c917c87934badb2bb4bc8c767a14e3a)

6 months ago[HUDI-2933] DISABLE Metadata table by default (#4213)
vinoth chandar [Sat, 4 Dec 2021 05:12:35 +0000 (21:12 -0800)] 
[HUDI-2933] DISABLE Metadata table by default (#4213)

(cherry picked from commit 0fd6b2d71e50a3b09336b74074a2f5117ba57f5c)

6 months ago[MINOR] Mitigate CI jobs timeout issues (#4173)
Raymond Xu [Sat, 4 Dec 2021 05:08:32 +0000 (21:08 -0800)] 
[MINOR] Mitigate CI jobs timeout issues (#4173)

* skip shutdown zookeeper in `@AfterAll` in TestHBaseIndex

* rebalance CI tests

(cherry picked from commit a799fae31610b2c38ca5aa52c87fa92363a41a70)

6 months agoRevert "[HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemVie...
zhangyue19921010 [Sat, 4 Dec 2021 02:56:53 +0000 (10:56 +0800)] 
Revert "[HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests"

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
(cherry picked from commit 5616830ae18ab3cbe1592d4e463a940342d25edc)

6 months ago[HUDI-2894][HUDI-2905] Metadata table - avoiding key lookup failures on base files...
Manoj Govindassamy [Fri, 3 Dec 2021 19:18:10 +0000 (11:18 -0800)] 
[HUDI-2894][HUDI-2905] Metadata table - avoiding key lookup failures on base files over S3 (#4185)

- Fetching partition files or all partitions from the metadata table is failing
   when run over S3. Metadata table uses HFile format for the base files and the
   record lookup uses HFile.Reader and HFileScanner interfaces to get records by
   partition keys. When the backing storage is S3, this record lookup from HFiles
   is failing with IOException, in turn failing the caller commit/update operations.

 - Metadata table looks up HFile records with positional read enabled so as to
   perform better for random lookups. But this positional read key lookup is
   returning with partial read sizes over S3 leading to HFile scanner throwing
   IOException. This doesn't happen over HDFS. Metadata table though uses the HFile
   for random key lookups, the positional read is not mandatory as we sort the keys
   when doing a lookup for multiple keys.

 - The fix is to disable HFile positional read for all HFile scanner based
   key lookups.

(cherry picked from commit 383d5edc169b79c4022f81a1580ab1bc2afebc30)

6 months agoRevert "[HUDI-2495] Resolve inconsistent key generation for timestamp types by Gener...
Yann Byron [Fri, 3 Dec 2021 16:13:38 +0000 (00:13 +0800)] 
Revert "[HUDI-2495] Resolve inconsistent key generation for timestamp types  by GenericRecord and Row (#3944)" (#4201)

(cherry picked from commit 2f96f4300b37207703b477979f2461bdd294ccf9)

6 months ago[HUDI-2911] Removing default value for `PARTITIONPATH_FIELD_NAME` resulting in incorr...
Alexey Kudinkin [Fri, 3 Dec 2021 12:33:38 +0000 (04:33 -0800)] 
[HUDI-2911] Removing default value for `PARTITIONPATH_FIELD_NAME` resulting in incorrect `KeyGenerator` configuration (#4195)

(cherry picked from commit bed7f9897a9127130c3d241df7634d44aa12167b)

6 months ago[HUDI-2902] Fixing populate meta fields with Hfile writers and Disabling virtual...
Sivabalan Narayanan [Fri, 3 Dec 2021 12:20:21 +0000 (07:20 -0500)] 
[HUDI-2902] Fixing populate meta fields with Hfile writers and Disabling virtual keys by default for metadata table (#4194)

(cherry picked from commit e483f7c776ed53015d08b78fc33729892dc2143c)

6 months ago[MINOR] use catalog schema if can not find table schema (#4182)
Yann Byron [Fri, 3 Dec 2021 08:37:13 +0000 (16:37 +0800)] 
[MINOR] use catalog schema if can not find table schema (#4182)

(cherry picked from commit ca427240c026bc6ba6e19b9b8b0a069a6330147a)

6 months ago[HUDI-2924] Refresh the fs view on successful checkpoints for write profile (#4199)
Danny Chan [Fri, 3 Dec 2021 08:12:59 +0000 (16:12 +0800)] 
[HUDI-2924] Refresh the fs view on successful checkpoints for write profile (#4199)

(cherry picked from commit 0699521f83eab843279ff7d78ac2b92fc80ad79b)

6 months ago[minor] Refactor write profile to always generate fs view (#4198)
Danny Chan [Fri, 3 Dec 2021 03:38:29 +0000 (11:38 +0800)] 
[minor] Refactor write profile to always generate fs view (#4198)

(cherry picked from commit f74b3d12aa89a96daafe43c69d37d816b9f4018b)

6 months ago[HUDI-2914] Fix remote timeline server config for flink (#4191)
Danny Chan [Fri, 3 Dec 2021 00:59:10 +0000 (08:59 +0800)] 
[HUDI-2914] Fix remote timeline server config for flink (#4191)

(cherry picked from commit 934fe54cc57b508875383cd807735b1323fef754)

6 months ago[HUDI-2904] Fix metadata table archival overstepping between regular writers and...
rmahindra123 [Thu, 2 Dec 2021 18:32:26 +0000 (10:32 -0800)] 
[HUDI-2904] Fix metadata table archival overstepping between regular writers and table services (#4186)

- Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
- Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>

(cherry picked from commit 91d2e61433e74abb44cb4d0ae236ee8f4a94e1f8)

6 months ago[MINOR] Fix the wrong usage of timestamp length variable bug (#4179)
zzzhy [Thu, 2 Dec 2021 14:47:31 +0000 (22:47 +0800)] 
[MINOR] Fix the wrong usage of timestamp length variable bug (#4179)

Signed-off-by: zzzhy <candle_1667@163.com>
(cherry picked from commit 61a03bc072c8d446a7334fdf7a284fca0a91f8c0)

6 months agoFixed partitions produced by layout optimization in case order-by key is composed...
Alexey Kudinkin [Thu, 2 Dec 2021 04:56:04 +0000 (20:56 -0800)] 
Fixed partitions produced by layout optimization in case order-by key is composed of a single column (#4183)

(cherry picked from commit 772f5ca24ee76c5c3676a9136fd994f7cd55a0e3)

6 months ago[HUDI-2881] Compact the file group with larger log files to reduce write amplificatio...
Shawy Geng [Thu, 2 Dec 2021 01:41:04 +0000 (09:41 +0800)] 
[HUDI-2881] Compact the file group with larger log files to reduce write amplification (#4152)

(cherry picked from commit 5284730175df4637eee43b179c774606b07a10a9)

6 months ago[HUDI-2880] Fixing loading of props from default dir (#4167)
Sivabalan Narayanan [Wed, 1 Dec 2021 08:02:30 +0000 (03:02 -0500)] 
[HUDI-2880] Fixing loading of props from default dir (#4167)

* Fixing loading of props from default dir

* addressing comments

(cherry picked from commit f4c25ba3fd11e2da1adcf2e089969a0b7f3dc62e)

6 months agoRevert "[HUDI-2856] Bit cask disk map delete modified (#4116)" (#4171)
Y Ethan Guo [Wed, 1 Dec 2021 06:08:44 +0000 (22:08 -0800)] 
Revert "[HUDI-2856] Bit cask disk map delete modified (#4116)" (#4171)

This reverts commit 257a6a74569cdb500b06c53b92d05f6f5833aa31.

(cherry picked from commit 9b254b6fc5779b6105b2fa9aeff5ad5b79407f84)

6 months agoRevert "[HUDI-2855] Change the default value of 'PAYLOAD_CLASS_NAME' to 'DefaultHoodi...
Alexey Kudinkin [Wed, 1 Dec 2021 01:47:16 +0000 (17:47 -0800)] 
Revert "[HUDI-2855] Change the default value of 'PAYLOAD_CLASS_NAME' to 'DefaultHoodieRecordPayload' (#4115)" (#4169)

This reverts commit 88067f57a23575aae3c371a7c7871e4207ca3bea.

(cherry picked from commit 24380c20606d63c7c129cb45a73f786f223e7d39)

6 months ago[HUDI-2891] Fix write configs for Java engine in Kafka Connect Sink (#4161)
Y Ethan Guo [Tue, 30 Nov 2021 14:45:50 +0000 (06:45 -0800)] 
[HUDI-2891] Fix write configs for Java engine in Kafka Connect Sink (#4161)

(cherry picked from commit ea009b55a385564f76beb99468f363edd3c4828a)

6 months ago[HUDI-2642] Add support ignoring case in update sql operation (#3882)
董可伦 [Tue, 30 Nov 2021 06:36:36 +0000 (14:36 +0800)] 
[HUDI-2642] Add support ignoring case in update sql operation (#3882)

(cherry picked from commit a398aad1fcbd5ff36eb7bc750fcc418d78b3a97e)

6 months ago[MINOR] Fix typo,rename 'getUrlEncodePartitoning' to 'getUrlEncodePartitioning' ...
董可伦 [Tue, 30 Nov 2021 02:31:22 +0000 (10:31 +0800)] 
[MINOR] Fix typo,rename 'getUrlEncodePartitoning' to 'getUrlEncodePartitioning' (#4130)

(cherry picked from commit 3433f00cb5366fa190a7b2d5254255041a705deb)

7 months agoBumping release candidate number 2 release-0.10.0-rc2
yuzhao.cyz [Mon, 29 Nov 2021 06:30:03 +0000 (14:30 +0800)] 
Bumping release candidate number 2

7 months ago[MINOR] Fix syntax error in create_source_release.sh (#4150)
Danny Chan [Mon, 29 Nov 2021 06:17:24 +0000 (14:17 +0800)] 
[MINOR] Fix syntax error in create_source_release.sh (#4150)

(cherry picked from commit 536af4b954abaa0d92a313bd5827173a232c6700)

7 months agoRemoving rfc from release package and fixing release validation script (#4147)
Sivabalan Narayanan [Mon, 29 Nov 2021 05:18:35 +0000 (00:18 -0500)] 
Removing rfc from release package and fixing release validation script (#4147)

(cherry picked from commit 38e75ea80681111f0b5cac753d738ce7a39bf6ac)

7 months ago[MINOR] Fixing integ test suite for hudi-aws and archival validation (#4142)
Sivabalan Narayanan [Mon, 29 Nov 2021 01:11:50 +0000 (20:11 -0500)] 
[MINOR] Fixing integ test suite for hudi-aws and archival validation (#4142)

(cherry picked from commit 52aae36b53e22bd7dab8c024ca64f40a1ade13be)

7 months ago[MINOR] fix typo (#4140)
vortual [Sun, 28 Nov 2021 09:13:50 +0000 (17:13 +0800)] 
[MINOR] fix typo (#4140)

(cherry picked from commit eca16932884d51f01ff33d9cd4c9d54cb20d7726)

7 months agoCreate release branch for version 0.10.0. release-0.10.0-rc1
yuzhao.cyz [Sat, 27 Nov 2021 09:22:39 +0000 (17:22 +0800)] 
Create release branch for version 0.10.0.

7 months ago[HUDI-2102] Support hilbert curve for hudi (#3952)
xiarixiaoyao [Sat, 27 Nov 2021 07:20:19 +0000 (15:20 +0800)] 
[HUDI-2102] Support hilbert curve for hudi (#3952)

Co-authored-by: Y Ethan Guo <ethan.guoyihua@gmail.com>
7 months ago[HUDI-2475] [HUDI-2862] Metadata table creation and avoid bootstrapping race for...
Manoj Govindassamy [Sat, 27 Nov 2021 07:19:26 +0000 (23:19 -0800)] 
[HUDI-2475] [HUDI-2862] Metadata table creation and avoid bootstrapping race for write client & add locking for upgrade (#4114)

Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
7 months ago[HUDI-2868] Fix skipped HoodieSparkSqlWriterSuite (#4125)
Raymond Xu [Sat, 27 Nov 2021 03:59:20 +0000 (19:59 -0800)] 
[HUDI-2868] Fix skipped HoodieSparkSqlWriterSuite (#4125)

- Co-authored-by: Yann Byron <biyan900116@gmail.com>

7 months ago[MINOR] Follow ups from HUDI-2861 (re-use same rollback instant for failed rollback...
Sivabalan Narayanan [Sat, 27 Nov 2021 00:22:53 +0000 (19:22 -0500)] 
[MINOR] Follow ups from HUDI-2861 (re-use same rollback instant for failed rollback) (#4133)

7 months ago[HUDI-2856] Bit cask disk map delete modified (#4116)
xuzifu666 [Fri, 26 Nov 2021 23:11:01 +0000 (07:11 +0800)] 
[HUDI-2856] Bit cask disk map delete modified (#4116)

* modified BitCaskDiskMap_close_function

* change iterators location to finally

* Update BitCaskDiskMap.java

7 months ago[HUDI-2864] Fix README and scripts with current limitations of hive sync (#4129)
rmahindra123 [Fri, 26 Nov 2021 23:09:32 +0000 (15:09 -0800)] 
[HUDI-2864] Fix README and scripts with current limitations of hive sync (#4129)

* Fix README with current limitations of hive sync

* Fix README with current limitations of hive sync

* Fix dep issue

* Fix Copy on Write flow

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
7 months ago[HUDI-2848] Excluse guava from hudi-cli pom (#4100)
huleilei [Fri, 26 Nov 2021 21:56:03 +0000 (05:56 +0800)] 
[HUDI-2848] Excluse guava from hudi-cli pom (#4100)

7 months ago[HUDI-2845] Metadata CLI - files/partition file listing fix and new validate option...
Manoj Govindassamy [Fri, 26 Nov 2021 21:44:16 +0000 (13:44 -0800)] 
[HUDI-2845] Metadata CLI - files/partition file listing fix and new validate option (#4092)

- Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>

7 months ago[HUDI-2767] Enabling timeline-server-based marker as default (#4112)
Y Ethan Guo [Fri, 26 Nov 2021 21:41:05 +0000 (13:41 -0800)] 
[HUDI-2767] Enabling timeline-server-based marker as default (#4112)

- Changes the default config of marker type (HoodieWriteConfig.MARKERS_TYPE or hoodie.write.markers.type) from DIRECT to TIMELINE_SERVER_BASED for Spark Engine.
- Adds engine-specific marker type configs: Spark -> TIMELINE_SERVER_BASED, Flink -> DIRECT, Java -> DIRECT.
- Uses DIRECT markers as well for Spark structured streaming due to timeline server only available for the first mini-batch.
- Fixes the marker creation method for non-partitioned table in TimelineServerBasedWriteMarkers.
- Adds the fallback to direct markers even when TIMELINE_SERVER_BASED is configured, in WriteMarkersFactory: when HDFS is used, or embedded timeline server is disabled, the fallback to direct markers happens.
- Fixes the closing of timeline service.
- Fixes tests that depend on markers, mainly by starting the timeline service for each test.

7 months ago[HUDI-2861] Re-use same rollback instant time for failed rollbacks (#4123)
Sivabalan Narayanan [Fri, 26 Nov 2021 21:36:42 +0000 (16:36 -0500)] 
[HUDI-2861] Re-use same rollback instant time for failed rollbacks (#4123)

7 months ago[MINOR] Fixing test failure to fix CI build failure (#4132)
Sivabalan Narayanan [Fri, 26 Nov 2021 18:50:10 +0000 (13:50 -0500)] 
[MINOR] Fixing test failure to fix CI build failure (#4132)

7 months ago[HUDI-2814] Addressing issues w/ Z-order Layout Optimization (#4060)
Alexey Kudinkin [Fri, 26 Nov 2021 18:02:15 +0000 (10:02 -0800)] 
[HUDI-2814] Addressing issues w/ Z-order Layout Optimization (#4060)

* `ZCurveOptimizeHelper` > `ZOrderingIndexHelper`;
Moved Z-index helper under `hudi.index.zorder` package

* Tidying up `ZOrderingIndexHelper`

* Fixing compilation

* Fixed index new/original table merging sequence to always prefer values from new index;
Cleaned up `HoodieSparkUtils`

* Added test for `mergeIndexSql`

* Abstracted Z-index name composition w/in `ZOrderingIndexHelper`;

* Fixed `DataSkippingUtils` to interrupt prunning in case data filter contains non-indexed column reference

* Properly handle exceptions origination during pruning in `HoodieFileIndex`

* Make sure no errors are logged upon encountering `AnalysisException`

* Cleaned up Z-index updating sequence;
Tidying up comments, java-docs;

* Fixed Z-index to properly handle changes of the list of clustered columns

* Tidying up

* `lint`

* Suppressing `JavaDocStyle` first sentence check

* Fixed compilation

* Fixing incorrect `DecimalType` conversion

* Refactored test `TestTableLayoutOptimization`
  - Added Z-index table composition test (against fixtures)
  - Separated out GC test;
Tidying up

* Fixed tests re-shuffling column order for Z-Index table `DataFrame` to align w/ the one by one loaded from JSON

* Scaffolded `DataTypeUtils` to do basic checks of Spark types;
Added proper compatibility checking b/w old/new index-tables

* Added test for Z-index tables merging

* Fixed import being shaded by creating internal `hudi.util` package

* Fixed packaging for `TestOptimizeTable`

* Revised `updateMetadataIndex` seq to provide Z-index updating process w/ source table schema

* Make sure existing Z-index table schema is sync'd to source table's one

* Fixed shaded refs

* Fixed tests

* Fixed type conversion of Parquet provided metadata values into Spark expected schemas

* Fixed `composeIndexSchema` utility to propose proper schema

* Added more tests for Z-index:
  - Checking that Z-index table is built correctly
  - Checking that Z-index tables are merged correctly (during update)

* Fixing source table

* Fixing tests to read from Parquet w/ proper schema

* Refactored `ParquetUtils` utility reading stats from Parquet footers

* Fixed incorrect handling of Decimals extracted from Parquet footers

* Worked around issues in javac failign to compile stream's collection

* Fixed handling of `Date` type

* Fixed handling of `DateType` to be parsed as `LocalDate`

* Updated fixture;
Make sure test loads Z-index fixture using proper schema

* Removed superfluous scheme adjusting when reading from Parquet, since Spark is actually able to perfectly restore schema (given Parquet was previously written by Spark as well)

* Fixing race-condition in Parquet's `DateStringifier` trying to share `SimpleDataFormat` object which is inherently not thread-safe

* Tidying up

* Make sure schema is used upon reading to validate input files are in the appropriate format;
Tidying up;

* Worked around javac (1.8) inability to infer expression type properly

* Updated fixtures;
Tidying up

* Fixing compilation after rebase

* Assert clustering have in Z-order layout optimization testing

* Tidying up exception messages

* XXX

* Added test validating Z-index lookup filter correctness

* Added more test-cases;
Tidying up

* Added tests for string expressions

* Fixed incorrect Z-index filter lookup translations

* Added more test-cases

* Added proper handling on complex negations of AND/OR expressions by pushing NOT operator down into inner expressions for appropriate handling

* Added `-target:jvm-1.8` for `hudi-spark` module

* Adding more tests

* Added tests for non-indexed columns

* Properly handle non-indexed columns by falling back to a re-write of containing expression as  `TrueLiteral` instead

* Fixed tests

* Removing the parquet test files and disabling corresponding tests

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
7 months ago[HUDI-2850] Fixing Clustering CLI - schedule and run command fixes to avoid NumberFor...
Manoj Govindassamy [Fri, 26 Nov 2021 12:17:23 +0000 (04:17 -0800)] 
[HUDI-2850] Fixing Clustering CLI - schedule and run command fixes to avoid NumberFormatException (#4101)

7 months ago[HUDI-2863] Rename option 'hoodie.parquet.page.size' to 'write.parquet.page.size...
Danny Chan [Fri, 26 Nov 2021 08:40:53 +0000 (16:40 +0800)] 
[HUDI-2863] Rename option 'hoodie.parquet.page.size' to 'write.parquet.page.size' (#4128)

7 months ago[HUDI-2852] Table metadata returns empty for non-exist partition (#4117)
mincwang [Fri, 26 Nov 2021 08:24:03 +0000 (16:24 +0800)] 
[HUDI-2852] Table metadata returns empty for non-exist partition (#4117)

* [HUDI-2852] Table metadata returns empty for non-exist partition

* add unit test

* fix code checkstyle

Co-authored-by: wangminchao <wangminchao@asinking.com>
7 months ago[MINOR] Include hudi-aws in flink bundle jar (#4127)
Danny Chan [Fri, 26 Nov 2021 06:36:44 +0000 (14:36 +0800)] 
[MINOR] Include hudi-aws in flink bundle jar (#4127)

HUDI-2801 makes this jar as required.

7 months ago[HUDI-2851] Shade org.apache.hadoop.hive.ql.optimizer package for flink bundle jar...
Ron [Fri, 26 Nov 2021 03:27:21 +0000 (11:27 +0800)] 
[HUDI-2851] Shade org.apache.hadoop.hive.ql.optimizer package for flink bundle jar (#4104)

7 months ago[HUDI-2005] Removing direct fs call in HoodieLogFileReader (#3865)
Sivabalan Narayanan [Thu, 25 Nov 2021 23:51:38 +0000 (18:51 -0500)] 
[HUDI-2005] Removing direct fs call in HoodieLogFileReader (#3865)

7 months ago[HUDI-2840] Fixed DeltaStreaemer to properly respect configuration passed t/h propert...
Alexey Kudinkin [Thu, 25 Nov 2021 22:48:22 +0000 (14:48 -0800)] 
[HUDI-2840] Fixed DeltaStreaemer to properly respect configuration passed t/h properties file (#4090)

* Rebased `DFSPropertiesConfiguration` to access Hadoop config in liue of FS to avoid confusion

* Fixed `readConfig` to take Hadoop's `Configuration` instead of FS;
Fixing usages

* Added test for local FS access

* Rebase to use `FSUtils.getFs`

* Combine properties provided as a file along w/ overrides provided from the CLI

* Added helper utilities to `HoodieClusteringConfig`;
Make sure corresponding config methods fallback to defaults;

* Fixed DeltaStreamer usage to respect properly combined configuration;
Abstracted `HoodieClusteringConfig.from` convenience utility to init Clustering config from `Properties`

* Tidying up

* `lint`

* Reverting changes to `HoodieWriteConfig`

* Tdiying up

* Fixed incorrect merge of the props

* Converted `HoodieConfig` to wrap around `Properties` into `TypedProperties`

* Fixed compilation

* Fixed compilation

7 months ago[HUDI-2801] Add Amazon CloudWatch metrics reporter (#4081)
Udit Mehrotra [Thu, 25 Nov 2021 21:33:16 +0000 (13:33 -0800)] 
[HUDI-2801] Add Amazon CloudWatch metrics reporter (#4081)

7 months ago[HUDI-2841] Fixing lazy rollback for MOR with list based strategy (#4110)
Sivabalan Narayanan [Thu, 25 Nov 2021 21:06:04 +0000 (16:06 -0500)] 
[HUDI-2841] Fixing lazy rollback for MOR with list based strategy (#4110)

7 months ago[HUDI-2858] Fixing handling of cluster update reject exception in deltastreamer ...
Sivabalan Narayanan [Thu, 25 Nov 2021 19:34:07 +0000 (14:34 -0500)] 
[HUDI-2858] Fixing handling of cluster update reject exception in deltastreamer (#4120)

7 months ago[HUDI-2671] Making error -> warn logs from timeline server with concurrent writers...
Sivabalan Narayanan [Thu, 25 Nov 2021 19:21:32 +0000 (14:21 -0500)] 
[HUDI-2671] Making error -> warn logs from timeline server with concurrent writers for inconsistent state (#4088)

* Making error -> warn logs from timeline server with concurrent writers for inconsistent state

* Fixing bad request response exception for timeline out of sync

* Addressing feedback. removed write concurrency mode depedency

7 months ago[HUDI-2794] Guarding table service commits within a single lock to commit to both...
Sivabalan Narayanan [Thu, 25 Nov 2021 19:19:30 +0000 (14:19 -0500)] 
[HUDI-2794] Guarding table service commits within a single lock to commit to both data table and metadata table (#4037)

* Fixing a single lock to commit table services across metadata table and data table

* Addressing comments

* rebasing with master

7 months ago[HUDI-2800] Remove rdd.isEmpty() validation to prevent CreateHandle being called...
Sagar Sumit [Thu, 25 Nov 2021 18:16:36 +0000 (23:46 +0530)] 
[HUDI-2800] Remove rdd.isEmpty() validation to prevent CreateHandle being called twice (#4121)

7 months ago[HUDI-1290] fixing mysql debezium source (#4119)
satishm [Thu, 25 Nov 2021 16:26:59 +0000 (21:56 +0530)] 
[HUDI-1290] fixing mysql debezium source (#4119)

7 months ago[HUDI-2480] FileSlice after pending compaction-requested instant-time… (#3703)
Danny Chan [Thu, 25 Nov 2021 14:30:09 +0000 (22:30 +0800)] 
[HUDI-2480] FileSlice after pending compaction-requested instant-time… (#3703)

* [HUDI-2480] FileSlice after pending compaction-requested instant-time is ignored by MOR snapshot reader

* include file slice after a pending compaction for spark reader

Co-authored-by: garyli1019 <yanjia.gary.li@gmail.com>
7 months ago[HUDI-2855] Change the default value of 'PAYLOAD_CLASS_NAME' to 'DefaultHoodieRecordP...
董可伦 [Thu, 25 Nov 2021 11:17:38 +0000 (19:17 +0800)] 
[HUDI-2855] Change the default value of 'PAYLOAD_CLASS_NAME' to 'DefaultHoodieRecordPayload' (#4115)

7 months ago[HUDI-2792] Configure metadata payload consistency check (#4035)
Sivabalan Narayanan [Thu, 25 Nov 2021 02:56:31 +0000 (21:56 -0500)] 
[HUDI-2792] Configure metadata payload consistency check (#4035)

- Relax metadata payload consistency check to consider spark task failures with spurious deletes

7 months ago[HUDI-1290] Add Debezium Source for deltastreamer (#4063)
rmahindra123 [Thu, 25 Nov 2021 01:57:02 +0000 (17:57 -0800)] 
[HUDI-1290] Add Debezium Source for deltastreamer (#4063)

* add source for postgres debezium

* Add tests for debezium payload

* Fix test

* Fix test

* Add tests for debezium source

* Add tests for debezium source

* Fix schema for debezium

* Fix checkstyle issues

* Fix config issue for schema registry

* Add mysql source for debezium

* Fix checkstyle issues an tests

* Improve code for merging toasted values

* Improve code for merging toasted values

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
7 months ago[HUDI-1290] [RFC-39] Deltastreamer avro source for Debezium CDC (#4048)
rmahindra123 [Thu, 25 Nov 2021 01:31:34 +0000 (17:31 -0800)] 
[HUDI-1290] [RFC-39] Deltastreamer avro source for Debezium CDC (#4048)

* Add RFC entry for deltastreamer source for debezium

* Add RFC for debezium source

* Add RFC for debezium source

* Add RFC for debezium source

* fix hyperlink issue and rebase

* Update progress

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
7 months ago[MINOR] Fix build failure due to checkstyle issues (#4111)
Y Ethan Guo [Thu, 25 Nov 2021 01:17:46 +0000 (17:17 -0800)] 
[MINOR] Fix build failure due to checkstyle issues (#4111)

7 months ago[HUDI-2844][CLI] Fixing archived Timeline crashing if timeline contains REPLACE_COMMI...
Alexey Kudinkin [Thu, 25 Nov 2021 00:53:29 +0000 (16:53 -0800)] 
[HUDI-2844][CLI] Fixing archived Timeline crashing if timeline contains REPLACE_COMMIT (#4091)

7 months ago[HUDI-2853] Add JMX deps in hudi utilities and kafka connect bundles (#4108)
rmahindra123 [Thu, 25 Nov 2021 00:03:01 +0000 (16:03 -0800)] 
[HUDI-2853] Add JMX deps in hudi utilities and kafka connect bundles (#4108)

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
7 months ago[HUDI-2793] Fixing deltastreamer checkpoint fetch/copy over (#4034)
Sivabalan Narayanan [Wed, 24 Nov 2021 23:26:40 +0000 (18:26 -0500)] 
[HUDI-2793] Fixing deltastreamer checkpoint fetch/copy over (#4034)

- Removed the copy over logic in transaction utils. Deltastreamer will go back to previous commits and get the checkpoint value.

7 months ago[HUDI-2766] Cluster update strategy should not be fenced by write config (#4093)
Sagar Sumit [Wed, 24 Nov 2021 18:15:40 +0000 (23:45 +0530)] 
[HUDI-2766] Cluster update strategy should not be fenced by write config (#4093)

Fix pending clustering rollback test

7 months ago[HUDI-2788] Fixing issues w/ Z-order Layout Optimization (#4026)
Alexey Kudinkin [Wed, 24 Nov 2021 18:10:28 +0000 (10:10 -0800)] 
[HUDI-2788] Fixing issues w/ Z-order Layout Optimization (#4026)

* Simplyfying, tidying up

* Fixed packaging for `TestOptimizeTable`

* Cleaned up `HoodiFileIndex` file filtering seq;
Removed optimization manually reading Parquet table circumventing Spark

* Refactored `DataSkippingUtils`:
  - Fixed checks to validate all statistics cols are present
  - Fixed some predicates being constructed incorrectly
  - Rewrote comments for easier comprehension, added more notes
  - Tidying up

* Tidying up tests

* `lint`

* Fixing compilation

* `TestOptimizeTable` > `TestTableLayoutOptimization`;
Added assertions to test data skipping paths

* Fixed tests to properly hit data-skipping path

* Fixed pruned files candidates lookup seq to conservatively included all non-indexed files

* Added java-doc

* Fixed compilation

7 months ago[HUDI-2443] Hudi KVComparator for all HFile writer usages (#3889)
Manoj Govindassamy [Wed, 24 Nov 2021 18:05:36 +0000 (10:05 -0800)] 
[HUDI-2443] Hudi KVComparator for all HFile writer usages (#3889)

* [HUDI-2443] Hudi KVComparator for all HFile writer usages

- Hudi relies on custom class shading for Hbase's KeyValue.KVComparator to
  avoid versioning and class loading issues. There are few places which are
  still using the Hbase's comparator class directly and version upgrades
  would make them obsolete. Refactoring the HoodieKVComparator and making
  all HFile writer creation using the same shaded class.

* [HUDI-2443] Hudi KVComparator for all HFile writer usages

- Moving HoodieKVComparator from common.bootstrap.index to common.util

* [HUDI-2443] Hudi KVComparator for all HFile writer usages

- Retaining the old HoodieKVComparatorV2 for boostrap case. Adding the
  new comparator as HoodieKVComparatorV2 to differentiate from the old
  one.

* [HUDI-2443] Hudi KVComparator for all HFile writer usages

 - Renamed HoodieKVComparatorV2 to HoodieMetadataKVComparator and moved it
   under the package org.apache.hudi.metadata.

* Make comparator classname configurable

* Revert new config and address other review comments

Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
7 months ago[HUDI-2671] Fix kafka offset handling in Kafka Connect protocol (#4021)
rmahindra123 [Wed, 24 Nov 2021 18:03:58 +0000 (10:03 -0800)] 
[HUDI-2671] Fix kafka offset handling in Kafka Connect protocol (#4021)

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
7 months ago[HUDI-2688] Claim the next rfc 40 for Hudi connector for Trino (#4105)
Sagar Sumit [Wed, 24 Nov 2021 16:43:37 +0000 (22:13 +0530)] 
[HUDI-2688] Claim the next rfc 40 for Hudi connector for Trino (#4105)

7 months ago[HUDI-2759] extract HoodieCatalogTable to coordinate spark catalog table and hoodie...
Yann Byron [Wed, 24 Nov 2021 10:12:38 +0000 (18:12 +0800)] 
[HUDI-2759] extract HoodieCatalogTable to coordinate spark catalog table and hoodie table (#3998)

7 months ago[HUDI-2847] Flink metadata table supports virtual keys (#4096)
Danny Chan [Wed, 24 Nov 2021 09:34:42 +0000 (17:34 +0800)] 
[HUDI-2847] Flink metadata table supports virtual keys (#4096)

7 months agoRevert "[HUDI-2799] Fix the classloader of flink write task (#4042)" (#4069)
Danny Chan [Wed, 24 Nov 2021 04:01:18 +0000 (12:01 +0800)] 
Revert "[HUDI-2799] Fix the classloader of flink write task (#4042)" (#4069)

This reverts commit 8281cbf7624c3a4eb90bf58671daf76843d00819.

7 months ago[HUDI-2838] refresh table after drop partition (#4084)
Yann Byron [Wed, 24 Nov 2021 03:46:48 +0000 (11:46 +0800)] 
[HUDI-2838] refresh table after drop partition (#4084)

7 months ago[HUDI-2818] Fix 2to3 upgrade when set `hoodie.table.keygenerator.class` (#4077)
Raymond Xu [Wed, 24 Nov 2021 03:43:34 +0000 (19:43 -0800)] 
[HUDI-2818] Fix 2to3 upgrade when set `hoodie.table.keygenerator.class` (#4077)

7 months ago[HUDI-2831] Securing usages of `SimpleDateFormat` to be thread-safe (#4073)
Alexey Kudinkin [Wed, 24 Nov 2021 01:25:11 +0000 (17:25 -0800)] 
[HUDI-2831] Securing usages of `SimpleDateFormat` to be thread-safe (#4073)

7 months ago[HUDI-2325] Add hive sync support to kafka connect (#3660)
rmahindra123 [Tue, 23 Nov 2021 23:48:06 +0000 (15:48 -0800)] 
[HUDI-2325] Add hive sync support to kafka connect (#3660)

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
7 months ago[MINOR] Fix typo,rename 'HooodieAvroDeserializer' to 'HoodieAvroDeserializer' (#4064)
董可伦 [Tue, 23 Nov 2021 11:10:57 +0000 (19:10 +0800)] 
[MINOR] Fix typo,rename 'HooodieAvroDeserializer' to 'HoodieAvroDeserializer' (#4064)

7 months ago[HUDI-2332] Add clustering and compaction in Kafka Connect Sink (#3857)
Y Ethan Guo [Tue, 23 Nov 2021 08:53:28 +0000 (00:53 -0800)] 
[HUDI-2332] Add clustering and compaction in Kafka Connect Sink (#3857)

* [HUDI-2332] Add clustering and compaction in Kafka Connect Sink

* Disable validation check on instant time for compaction and adjust configs

* Add javadocs

* Add clustering and compaction config

* Fix transaction causing missing records in the target table

* Add debugging logs

* Fix kafka offset sync in participant

* Adjust how clustering and compaction are configured in kafka-connect

* Fix clustering strategy

* Remove irrelevant changes from other published PRs

* Update clustering logic and others

* Update README

* Fix test failures

* Fix indentation

* Fix clustering config

* Add JavaCustomColumnsSortPartitioner and make async compaction enabled by default

* Add test for JavaCustomColumnsSortPartitioner

* Add more changes after IDE sync

* Update README with clarification

* Fix clustering logic after rebasing

* Remove unrelated changes

7 months ago[HUDI-2409] Using HBase shaded jars in Hudi presto bundle (#3623)
zhangyue19921010 [Tue, 23 Nov 2021 05:55:12 +0000 (13:55 +0800)] 
[HUDI-2409] Using HBase shaded jars in Hudi presto bundle (#3623)

* using hbase-shaded-jars-in-hudi-presto-hundle

* test

* add hudi-common-bundle

* code review

* code review

* code review

* code review

* test

* test

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
7 months ago[HUDI-2778] Optimize statistics collection related codes and add some docs for z...
xiarixiaoyao [Tue, 23 Nov 2021 05:46:02 +0000 (13:46 +0800)] 
[HUDI-2778] Optimize statistics collection related codes and add some docs for z-order add fix some bugs (#4013)

* [HUDI-2778] Optimize statistics collection related codes and add more docs for z-order.

* add test code for multi-thread parquet footer read

7 months ago[HUDI-2743] Assume path exists and defer fs.exists() in AbstractTableFileSystemView...
Sagar Sumit [Tue, 23 Nov 2021 03:13:10 +0000 (08:43 +0530)] 
[HUDI-2743] Assume path exists and defer fs.exists() in AbstractTableFileSystemView (#4002)

7 months ago[MINOR] Add more configuration to Kafka setup script (#3992)
Y Ethan Guo [Tue, 23 Nov 2021 02:03:38 +0000 (18:03 -0800)] 
[MINOR] Add more configuration to Kafka setup script (#3992)

* [MINOR] Add more configuration to Kafka setup script

* Add option to reuse Kafka topic

* Minor fixes to README

7 months ago[HUDI-1937] Rollback unfinished replace commit to allow updates (#3869)
Sagar Sumit [Tue, 23 Nov 2021 01:59:03 +0000 (07:29 +0530)] 
[HUDI-1937] Rollback unfinished replace commit to allow updates (#3869)

* [HUDI-1937] Rollback unfinished replace commit to allow updates while clustering

* Revert and delete requested replacecommit too

* Rollback pending clustering instants transactionally

* No double locking and add a config to enable rollback

* Update config to be clear about rollback only on conflict

7 months ago[MINOR] Fix typo,'multipe' corrected to 'multiple' (#4068)
Jimmy.Zhou [Tue, 23 Nov 2021 01:20:23 +0000 (09:20 +0800)] 
[MINOR] Fix typo,'multipe' corrected to 'multiple' (#4068)

7 months ago[HUDI-2737] Use earliest instant by default for async compaction and clustering jobs...
Y Ethan Guo [Tue, 23 Nov 2021 01:19:41 +0000 (17:19 -0800)] 
[HUDI-2737] Use earliest instant by default for async compaction and clustering jobs (#3991)

Address review comments

Fix test failures

Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
7 months ago[HUDI-2550] Expand File-Group candidates list for appending for MOR tables (#3986)
Alexey Kudinkin [Tue, 23 Nov 2021 00:19:59 +0000 (16:19 -0800)] 
[HUDI-2550] Expand File-Group candidates list for appending for MOR tables (#3986)

7 months ago[HUDI-2599] Make addFilesToview and fetchLatestBaseFiles public (#4066)
Sagar Sumit [Mon, 22 Nov 2021 17:23:50 +0000 (22:53 +0530)] 
[HUDI-2599] Make addFilesToview and fetchLatestBaseFiles public (#4066)

7 months ago[HUDI-2559] Converting commit timestamp format to millisecs (#4024)
Sivabalan Narayanan [Mon, 22 Nov 2021 16:44:38 +0000 (11:44 -0500)] 
[HUDI-2559] Converting commit timestamp format to millisecs (#4024)

- Adds support for generating commit timestamps with millisecs granularity.
- Older commit timestamps (in secs granularity) will be suffixed with 999 and parsed with millisecs format.

7 months ago[MINOR] Fix instant parsing in HoodieClusteringJob (#4071)
Sagar Sumit [Mon, 22 Nov 2021 13:57:44 +0000 (19:27 +0530)] 
[MINOR] Fix instant parsing in HoodieClusteringJob (#4071)

7 months ago[HUDI-2472] Enabling metadata table for TestHoodieIndex test case (#4045)
Manoj Govindassamy [Mon, 22 Nov 2021 12:21:24 +0000 (04:21 -0800)] 
[HUDI-2472] Enabling metadata table for TestHoodieIndex test case (#4045)

- Enablng the metadata table for testSimpleGlobalIndexTagLocationWhenShouldUpdatePartitionPath.
   This is more of a test issue.

7 months ago[HUDI-2533] New option for hoodieClusteringJob to check, rollback and re-execute...
zhangyue19921010 [Mon, 22 Nov 2021 11:00:33 +0000 (19:00 +0800)] 
[HUDI-2533] New option for hoodieClusteringJob to check, rollback and re-execute the last failed clustering job (#3765)

* coding finished and need to do uts

* add uts

* code review

* code review

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
7 months ago[HUDI-1870] Add more Spark CI build tasks (#4022)
Raymond Xu [Mon, 22 Nov 2021 10:16:45 +0000 (02:16 -0800)] 
[HUDI-1870] Add more Spark CI build tasks  (#4022)

* [HUDI-1870] Add more Spark CI build tasks

- build for spark3.0.x
- build for spark-shade-unbundle-avro
- fix build failures
  - delete unnecessary assertion for spark 3.0.x
  - use AvroConversionUtils#convertAvroSchemaToStructType instead of calling SchemaConverters#toSqlType directly to solve the compilation failures with spark-shade-unbundle-avro (#5)

Co-authored-by: Yann <biyan900116@gmail.com>
7 months ago[HUDI-2799] Fix the classloader of flink write task (#4042)
Danny Chan [Mon, 22 Nov 2021 03:05:05 +0000 (11:05 +0800)] 
[HUDI-2799] Fix the classloader of flink write task (#4042)

7 months ago[MINOR] Fix typos (#4053)
董可伦 [Sun, 21 Nov 2021 08:34:59 +0000 (16:34 +0800)] 
[MINOR] Fix typos (#4053)

7 months ago[HUDI-1932] Update Hive sync timestamp when change detected (#3053)
Nate Radtke [Sun, 21 Nov 2021 06:41:05 +0000 (00:41 -0600)] 
[HUDI-1932] Update Hive sync timestamp when change detected (#3053)

* Update Hive sync timestamp when change detected

Only update the last commit timestamp on the Hive table when the table schema
has changed or a partition is created/updated.

When using AWS Glue Data Catalog as the metastore for Hive this will ensure
that table versions are substantive (including schema and/or partition
changes). Prior to this change when a Hive sync is performed without schema
or partition changes the table in the Glue Data Catalog would have a new
version published with the only change being the timestamp property.

https://issues.apache.org/jira/browse/HUDI-1932

* add conditional sync flag

* fix testSyncWithoutDiffs

* fix HiveSyncConfig

Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>