projects / gobblin.git / commit
? search:
summary | shortlog | log | commit | commitdiff | tree
(parent: 23f085a) | patch
[GOBBLIN-1335] Publish GMCE(GobblinMetadataChangeEvent) publisher and iceberg retenti...
authorZihan Li <zihli@linkedin.com>
Thu, 4 Feb 2021 20:13:05 +0000 (12:13 -0800)
committerLei Sun <autumnust@gmail.com>
Thu, 4 Feb 2021 20:13:05 +0000 (12:13 -0800)
commit4e2dc94eb570086c768c247f669255d62f84f3d6
treedd04b2e282b8220aaf84628d3c8e9ae3d1811a03tree | snapshot
parent23f085a12e2bc628b1c25582f10c79d987a5f65acommit | diff
[GOBBLIN-1335] Publish GMCE(GobblinMetadataChangeEvent) publisher and iceberg retention job to Gobblin OSS

[GOBBLIN-1335] Publish
GMCE(GobblinMetadataChangeEvent) publisher and
iceberg retention job to Gobblin OSS

add OSS comments

add unit test

Add icebergWriter to be able to consume GMCE and
register into iceberg table

Make it re-writable for catalog inside
icebergWriter

address comments

trigger build

trigger build

address comments

Merge branch 'master' into GOBBLIN-1335

Merge branch 'master' into GOBBLIN-1335

define iceberg version

address comments

include recent changes

Merge remote-tracking branch 'origin' into
GOBBLIN-1335

Merge remote-tracking branch 'origin' into
GOBBLIN-1335

merge master and modify method privacy level to
make them be re-writable

Closes #3172 from ZihanLi58/GOBBLIN-1335
31 files changed:
conf/aws/application.conf diff | blob | history
gobblin-all/build.gradle diff | blob | history
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java diff | blob | history
gobblin-compaction/build.gradle diff | blob | history
gobblin-compaction/src/main/java/org/apache/gobblin/compaction/action/CompactionGMCEPublishingAction.java [new file with mode: 0644] blob
gobblin-compaction/src/test/java/org/apache/gobblin/compaction/action/CompactionGMCEPublishingActionTest.java [new file with mode: 0644] blob
gobblin-data-management/build.gradle diff | blob | history
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/retention/dataset/CleanableIcebergDataset.java [new file with mode: 0644] blob
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/retention/profile/ManagedIcebergCleanableDatasetFinder.java [new file with mode: 0644] blob
gobblin-iceberg/build.gradle [new file with mode: 0644] blob
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/GobblinMCEProducer.java [new file with mode: 0644] blob
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/Utils/IcebergUtils.java [new file with mode: 0644] blob
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/Utils/TypeInfoToSchemaParser.java [new file with mode: 0644] blob
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/publisher/GobblinMCEPublisher.java [new file with mode: 0644] blob
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/writer/GobblinMCEWriter.java [new file with mode: 0644] blob
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/writer/GobblinMCEWriterBuilder.java [new file with mode: 0644] blob
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/writer/IcebergMCEMetadataKeys.java [new file with mode: 0644] blob
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/writer/IcebergMetadataWriter.java [new file with mode: 0644] blob
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/writer/MetadataWriter.java [new file with mode: 0644] blob
gobblin-iceberg/src/test/java/org/apache/gobblin/iceberg/publisher/GobblinMCEPublisherTest.java [new file with mode: 0644] blob
gobblin-iceberg/src/test/java/org/apache/gobblin/iceberg/writer/GobblinMCEWriterTest.java [new file with mode: 0644] blob
gobblin-iceberg/src/test/resources/publisherTest/data.json [new file with mode: 0644] blob
gobblin-iceberg/src/test/resources/publisherTest/schema.avsc [new file with mode: 0644] blob
gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/DatasetIdentifier.avsc [new file with mode: 0644] blob
gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GobblinMetadataChangeEvent.avsc [new file with mode: 0644] blob
gobblin-modules/gobblin-kafka-common/build.gradle diff | blob | history
gradle/scripts/defaultBuildProperties.gradle diff | blob | history
gradle/scripts/dependencyDefinitions.gradle diff | blob | history
maven-nexus/upload-to-nexus.sh diff | blob | history
maven-sonatype/upload-to-sonatype.sh diff | blob | history
settings.gradle diff | blob | history
A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.