[GOBBLIN-1335] Publish GMCE(GobblinMetadataChangeEvent) publisher and iceberg retenti...
authorZihan Li <zihli@linkedin.com>
Thu, 4 Feb 2021 20:13:05 +0000 (12:13 -0800)
committerLei Sun <autumnust@gmail.com>
Thu, 4 Feb 2021 20:13:05 +0000 (12:13 -0800)
commit4e2dc94eb570086c768c247f669255d62f84f3d6
treedd04b2e282b8220aaf84628d3c8e9ae3d1811a03
parent23f085a12e2bc628b1c25582f10c79d987a5f65a
[GOBBLIN-1335] Publish GMCE(GobblinMetadataChangeEvent) publisher and iceberg retention job to Gobblin OSS

[GOBBLIN-1335] Publish
GMCE(GobblinMetadataChangeEvent) publisher and
iceberg retention job to Gobblin OSS

add OSS comments

add unit test

Add icebergWriter to be able to consume GMCE and
register into iceberg table

Make it re-writable for catalog inside
icebergWriter

address comments

trigger build

trigger build

address comments

Merge branch 'master' into GOBBLIN-1335

Merge branch 'master' into GOBBLIN-1335

define iceberg version

address comments

include recent changes

Merge remote-tracking branch 'origin' into
GOBBLIN-1335

Merge remote-tracking branch 'origin' into
GOBBLIN-1335

merge master and modify method privacy level to
make them be re-writable

Closes #3172 from ZihanLi58/GOBBLIN-1335
31 files changed:
conf/aws/application.conf
gobblin-all/build.gradle
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java
gobblin-compaction/build.gradle
gobblin-compaction/src/main/java/org/apache/gobblin/compaction/action/CompactionGMCEPublishingAction.java [new file with mode: 0644]
gobblin-compaction/src/test/java/org/apache/gobblin/compaction/action/CompactionGMCEPublishingActionTest.java [new file with mode: 0644]
gobblin-data-management/build.gradle
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/retention/dataset/CleanableIcebergDataset.java [new file with mode: 0644]
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/retention/profile/ManagedIcebergCleanableDatasetFinder.java [new file with mode: 0644]
gobblin-iceberg/build.gradle [new file with mode: 0644]
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/GobblinMCEProducer.java [new file with mode: 0644]
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/Utils/IcebergUtils.java [new file with mode: 0644]
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/Utils/TypeInfoToSchemaParser.java [new file with mode: 0644]
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/publisher/GobblinMCEPublisher.java [new file with mode: 0644]
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/writer/GobblinMCEWriter.java [new file with mode: 0644]
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/writer/GobblinMCEWriterBuilder.java [new file with mode: 0644]
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/writer/IcebergMCEMetadataKeys.java [new file with mode: 0644]
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/writer/IcebergMetadataWriter.java [new file with mode: 0644]
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/writer/MetadataWriter.java [new file with mode: 0644]
gobblin-iceberg/src/test/java/org/apache/gobblin/iceberg/publisher/GobblinMCEPublisherTest.java [new file with mode: 0644]
gobblin-iceberg/src/test/java/org/apache/gobblin/iceberg/writer/GobblinMCEWriterTest.java [new file with mode: 0644]
gobblin-iceberg/src/test/resources/publisherTest/data.json [new file with mode: 0644]
gobblin-iceberg/src/test/resources/publisherTest/schema.avsc [new file with mode: 0644]
gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/DatasetIdentifier.avsc [new file with mode: 0644]
gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GobblinMetadataChangeEvent.avsc [new file with mode: 0644]
gobblin-modules/gobblin-kafka-common/build.gradle
gradle/scripts/defaultBuildProperties.gradle
gradle/scripts/dependencyDefinitions.gradle
maven-nexus/upload-to-nexus.sh
maven-sonatype/upload-to-sonatype.sh
settings.gradle