beam.git
2 hours agoMerge pull request #17572: [BEAM-14428] I/O, community, and contribute pages improvements master
Alexey Romanenko [Thu, 19 May 2022 08:32:09 +0000 (10:32 +0200)] 
Merge pull request #17572: [BEAM-14428] I/O, community, and contribute pages improvements

15 hours agoUpdate release-guide.md nightly-refs/heads/master
Yichi Zhang [Wed, 18 May 2022 19:29:38 +0000 (12:29 -0700)] 
Update release-guide.md

18 hours agoBump github.com/lib/pq from 1.10.5 to 1.10.6 in /sdks (#17691)
dependabot[bot] [Wed, 18 May 2022 17:19:00 +0000 (10:19 -0700)] 
Bump github.com/lib/pq from 1.10.5 to 1.10.6 in /sdks (#17691)

Bumps [github.com/lib/pq](https://github.com/lib/pq) from 1.10.5 to 1.10.6.
- [Release notes](https://github.com/lib/pq/releases)
- [Commits](https://github.com/lib/pq/compare/v1.10.5...v1.10.6)

---
updated-dependencies:
- dependency-name: github.com/lib/pq
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
18 hours agoFix a few small linting bugs (#17695)
Danny McCormick [Wed, 18 May 2022 17:18:29 +0000 (13:18 -0400)] 
Fix a few small linting bugs (#17695)

19 hours ago[BEAM-10529] update KafkaIO Xlang integration test to publish and receive null keys...
johnjcasey [Wed, 18 May 2022 16:15:37 +0000 (12:15 -0400)] 
[BEAM-10529] update KafkaIO Xlang integration test to publish and receive null keys (#17319)

* [BEAM-10529] update test to publish and receive null keys

* [BEAM-10529] add test with a populated key to kafka xlang_kafkaio_it_test.py

20 hours ago[BEAM-14474] Suppress 'Mean of empty slice' Runtime Warning in dataframe unit test...
Yi Hu [Wed, 18 May 2022 15:23:21 +0000 (11:23 -0400)] 
[BEAM-14474] Suppress 'Mean of empty slice' Runtime Warning in dataframe unit test (#17682)

* [BEAM-14474] Suppress 'Mean of empty slice' Runtime Warning in dataframe unit test

* use catch_warnings context manager

42 hours ago [BEAM-14467] Fix bug where run_pytest.sh does not elevate errors raised in no_xdist...
Brian Hulette [Tue, 17 May 2022 16:49:13 +0000 (09:49 -0700)] 
 [BEAM-14467] Fix bug where run_pytest.sh does not elevate errors raised in no_xdist tests (#17687)

43 hours ago[BEAM-12000] Update programming-guide.md (#17679)
tvalentyn [Tue, 17 May 2022 16:16:53 +0000 (18:16 +0200)] 
[BEAM-12000] Update programming-guide.md (#17679)

44 hours agoREADME update for the Docker Error 255 during Website launch on Apple Silicon (#17456)
Darkhan Nausharipov [Tue, 17 May 2022 14:41:32 +0000 (20:41 +0600)] 
README update for the Docker Error 255 during Website launch on Apple Silicon (#17456)

Co-authored-by: Danny McCormick <dannymccormick@google.com>
45 hours ago[BEAM-14441] Automatically assign issue labels based on responses to template (#17661)
Danny McCormick [Tue, 17 May 2022 14:16:31 +0000 (09:16 -0500)] 
[BEAM-14441] Automatically assign issue labels based on responses to template (#17661)

45 hours agoMerge pull request #17692 from aromanenko-dev/BEAM-14035-checkstyle-fix
Moritz Mack [Tue, 17 May 2022 13:58:05 +0000 (15:58 +0200)] 
Merge pull request #17692 from aromanenko-dev/BEAM-14035-checkstyle-fix

[BEAM-14035] Fix checkstyle issue

46 hours ago[BEAM-14035] Fix checkstyle issue 17692/head
Alexey Romanenko [Tue, 17 May 2022 12:52:36 +0000 (14:52 +0200)] 
[BEAM-14035] Fix checkstyle issue

47 hours agoMerge pull request #17662:[BEAM-14334] Remove remaining forkEvery 1 from all Spark...
Alexey Romanenko [Tue, 17 May 2022 11:56:50 +0000 (13:56 +0200)] 
Merge pull request #17662:[BEAM-14334] Remove remaining forkEvery 1 from all Spark tests and stop mixing unit tests with runner validations.

2 days agoMerge pull request #17678 from [BEAM-14460] [Playground] WIP. Fix error during gettin...
Vladislav Chunikhin [Tue, 17 May 2022 09:05:58 +0000 (12:05 +0300)] 
Merge pull request #17678 from [BEAM-14460] [Playground] WIP. Fix error during getting the graph for java SDK.

2 days ago[BEAM-14411] Re-enable TypecodersTest, fix most issues (#17547)
Brian Hulette [Mon, 16 May 2022 22:55:35 +0000 (15:55 -0700)] 
[BEAM-14411] Re-enable TypecodersTest, fix most issues (#17547)

* Re-enable TypecodersTest, fix most issues

* yapf

* lint

2 days agoUpdate release-guide.md
Yichi Zhang [Mon, 16 May 2022 22:54:57 +0000 (15:54 -0700)] 
Update release-guide.md

2 days agoUpdate release-guide.md
Yichi Zhang [Mon, 16 May 2022 22:47:11 +0000 (15:47 -0700)] 
Update release-guide.md

2 days agoAdding eslint and lint configuration to TypeScript SDK (#17676)
David Huntsperger [Mon, 16 May 2022 19:34:43 +0000 (12:34 -0700)] 
Adding eslint and lint configuration to TypeScript SDK (#17676)

Added eslint and TypeScript linting config to the TypeScript SDK. You can run it with npm run lint, but it's not integrated with the build step, so won't fail the build -- just informational.

2 days agoMerge pull request #17520 from BEAM-12356 Close DatasetService leaked with getTable
Minbo Bae [Mon, 16 May 2022 19:32:17 +0000 (12:32 -0700)] 
Merge pull request #17520 from BEAM-12356 Close DatasetService leaked with getTable

2 days agoAdd Akvelon to case-studies (#17611)
bullet03 [Mon, 16 May 2022 19:01:24 +0000 (01:01 +0600)] 
Add Akvelon to case-studies (#17611)

* add image and md file to case-studies

* change text solid for deep

2 days ago[BEAM-14035 ] Implement BigQuerySchema Read/Write TransformProvider (#17607)
Damon Douglas [Mon, 16 May 2022 18:02:39 +0000 (18:02 +0000)] 
[BEAM-14035 ] Implement BigQuerySchema Read/Write TransformProvider (#17607)

2 days ago[BEAM-14440] Add basic fuzz tests to the coders package (#17587)
Jack McCluskey [Mon, 16 May 2022 17:46:20 +0000 (13:46 -0400)] 
[BEAM-14440] Add basic fuzz tests to the coders package (#17587)

* UTF8 fuzz test

* Ints fuzz tests

* Bytes fuzz tests

* Double fuzz test

* Move fuzz tests to standalone test file

2 days ago[BEAM-14473] Throw error if using globally windowed, unbounded side input (#17681)
Jack McCluskey [Mon, 16 May 2022 16:33:33 +0000 (12:33 -0400)] 
[BEAM-14473] Throw error if using globally windowed, unbounded side input (#17681)

2 days ago[BEAM-13982] Added output of logging for python E2E pytests (#17637)
Ryan Thompson [Mon, 16 May 2022 14:30:14 +0000 (10:30 -0400)] 
[BEAM-13982] Added output of logging for python E2E pytests (#17637)

2 days ago[BEAM-13015] Update the SDK harness grouping table to be memory bounded based upon...
Lukasz Cwik [Mon, 16 May 2022 14:08:44 +0000 (07:08 -0700)] 
[BEAM-13015] Update the SDK harness grouping table to be memory bounded based upon the amount of assigned cache memory and to use an LRU eviction policy. (#17327)

* [BEAM-13015] Update the grouping table to be memory bounded based upon amount of assigned cache memory and also use an LRU policy for evicting entries from the table.

* fixup! checkstyle

* fixup! Address PR comments.

4 days ago[BEAM-14470] Use Generic Registrations in loadtests. (#17673)
Robert Burke [Sat, 14 May 2022 22:49:04 +0000 (15:49 -0700)] 
[BEAM-14470] Use Generic Registrations in loadtests. (#17673)

5 days ago[BEAM-14371] (and BEAM-14372) - enable a couple staticchecks (#17670)
Danny McCormick [Fri, 13 May 2022 23:04:58 +0000 (19:04 -0400)] 
[BEAM-14371] (and BEAM-14372) - enable a couple staticchecks (#17670)

5 days agoAdd some auto-starting runners to the typescript SDK. (#17580)
Robert Bradshaw [Fri, 13 May 2022 22:00:42 +0000 (15:00 -0700)] 
Add some auto-starting runners to the typescript SDK. (#17580)

Adds out-of-the-box support for FlinkRunner, DataflowRunner, and the Python Universal Local runner. Also adds a DefaultRunner which chooses between the DirectRunner and the ULR depending on the properties of the pipeline.

5 days ago[BEAM-14469] Allow nil primary returns from TrySplit in a single-window context...
Jack McCluskey [Fri, 13 May 2022 21:20:49 +0000 (17:20 -0400)] 
[BEAM-14469] Allow nil primary returns from TrySplit in  a single-window context (#17667)

5 days ago[BEAM-14014] Support impersonation credentials in dataflow runner (#17244)
Ryan Thompson [Fri, 13 May 2022 19:28:22 +0000 (15:28 -0400)] 
[BEAM-14014] Support impersonation credentials in dataflow runner (#17244)

5 days agoMerge pull request #17605 from ihji/BEAM-14455
Heejong Lee [Fri, 13 May 2022 18:37:31 +0000 (11:37 -0700)] 
Merge pull request #17605 from ihji/BEAM-14455

[BEAM-14455] Add UUID to sub-schemas for PythonExternalTransform

5 days agoMerge pull request #17365 from [BEAM-12482] Update Schema Destination during Bigquery...
Marco Robles [Fri, 13 May 2022 17:50:53 +0000 (12:50 -0500)] 
Merge pull request #17365 from [BEAM-12482] Update Schema Destination during Bigquery load job when using temporary tables using zeroloadjob

* Added a zero row job to bigquery writeTable

* added unit test

* add sdf for update schema destination

* add loadjob with inputstreamcontent parameter

* add updateschemadest with zeroloadjob before copying to dest

* add test for write temp tables

* add nullness supress warning

* remove unnecesary variable and use global variable for updateschemadestination-dofn

Co-authored-by: Miguel Anzo <miguel.anzo@wizeline.com>
5 days agoMerge pull request #17608 from ihji/BEAM-14430
Heejong Lee [Fri, 13 May 2022 17:29:55 +0000 (10:29 -0700)] 
Merge pull request #17608 from ihji/BEAM-14430

[BEAM-14430] Adding a logical type support for Python callables to Row schema

5 days agoMerge pull request #17653 Revert "Better test assertion."
Robert Bradshaw [Fri, 13 May 2022 16:23:25 +0000 (09:23 -0700)] 
Merge pull request #17653 Revert "Better test assertion."

5 days ago[BEAM-14465] Reduce DefaultS3ClientBuilderFactory logging to debug level (#17645)
Jack McCluskey [Fri, 13 May 2022 16:16:31 +0000 (12:16 -0400)] 
[BEAM-14465] Reduce DefaultS3ClientBuilderFactory logging to debug level (#17645)

5 days agoBump github.com/spf13/cobra from 1.3.0 to 1.4.0 in /sdks (#17647)
dependabot[bot] [Fri, 13 May 2022 16:14:26 +0000 (09:14 -0700)] 
Bump github.com/spf13/cobra from 1.3.0 to 1.4.0 in /sdks (#17647)

Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.3.0 to 1.4.0.
- [Release notes](https://github.com/spf13/cobra/releases)
- [Changelog](https://github.com/spf13/cobra/blob/v1.4.0/CHANGELOG.md)
- [Commits](https://github.com/spf13/cobra/compare/v1.3.0...v1.4.0)

---
updated-dependencies:
- dependency-name: github.com/spf13/cobra
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
5 days ago[BEAM-14334] Remove remaining forkEvery 1 from all Spark tests and stop mixing unit... 17662/head
Moritz Mack [Fri, 13 May 2022 14:16:56 +0000 (16:16 +0200)] 
[BEAM-14334] Remove remaining forkEvery 1 from all Spark tests and stop mixing unit tests with runner validations.

6 days agoDrop dataclasses requirement, we only support python 3.7+ (#17640)
Brian Hulette [Fri, 13 May 2022 06:56:21 +0000 (23:56 -0700)] 
Drop dataclasses requirement, we only support python 3.7+ (#17640)

6 days agoRevert "Better test assertion. (#17551)" 17653/head
tvalentyn [Fri, 13 May 2022 06:54:11 +0000 (08:54 +0200)] 
Revert "Better test assertion. (#17551)"

This reverts commit dc20185f6af2e862bd62f9df7b835593a466c5f1.

6 days ago[BEAM-14347] Add function for simple function registration (#17650)
Danny McCormick [Fri, 13 May 2022 02:12:53 +0000 (22:12 -0400)] 
[BEAM-14347] Add function for simple function registration (#17650)

6 days agoadd comment 17608/head
Heejong Lee [Fri, 13 May 2022 01:02:09 +0000 (18:02 -0700)] 
add comment

6 days agoMerge pull request #17631 from chamikaramj/update_release_md
Ahmet Altay [Thu, 12 May 2022 23:34:49 +0000 (16:34 -0700)] 
Merge pull request #17631 from chamikaramj/update_release_md

6 days agoput a default type hint for PythonCallableSource
Heejong Lee [Thu, 12 May 2022 23:12:46 +0000 (16:12 -0700)] 
put a default type hint for PythonCallableSource

6 days agoBump cloud.google.com/go/bigquery from 1.28.0 to 1.32.0 in /sdks (#17625)
dependabot[bot] [Thu, 12 May 2022 22:25:59 +0000 (15:25 -0700)] 
Bump cloud.google.com/go/bigquery from 1.28.0 to 1.32.0 in /sdks (#17625)

Bumps [cloud.google.com/go/bigquery](https://github.com/googleapis/google-cloud-go) from 1.28.0 to 1.32.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.28.0...spanner/v1.32.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/bigquery
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
6 days agoadd micros_instant urn
Heejong Lee [Thu, 12 May 2022 22:10:50 +0000 (15:10 -0700)] 
add micros_instant urn

6 days agomove logical types def
Heejong Lee [Thu, 12 May 2022 21:46:24 +0000 (14:46 -0700)] 
move logical types def

6 days agoMerge pull request #17408 from [BEAM-14312] [Website] change section order, move...
bullet03 [Thu, 12 May 2022 20:48:41 +0000 (02:48 +0600)] 
Merge pull request #17408 from [BEAM-14312] [Website] change section order, move socials to footer

* [BEAM-14312] [Website] change section order, move socials to footer

* [BEAM-14312] [Website] change margin

* [BEAM-14312] [Website] change link

* [BEAM-14312] [Website] delete section, change title and links of the pillars

* [BEAM-14312] [Website] add pipelines section

* [BEAM-14312] [Website] change image in graphic section

* [BEAM-14312] [Website] delete width of 95%, change flink link

* [BEAM-14312] [Website] move css style, delete unused html, optimized logos size

* [BEAM-14312] [Website] substitute images, fix css styles for pipelines and logos sections

* [BEAM-14312] [Website] add kinesis runner

* [BEAM-14312] [Website] add kinesis runner, add margin-right to pipelines-logos

* [BEAM-14312] [Website] change amazon kinesis logo

* [BEAM-14312] change structure of logos in the footer

* [BEAM-14312] add license header to linkedin.svg

* [BEAM-14312] change margin in quote section

6 days agoBump cloud.google.com/go/pubsub from 1.18.0 to 1.21.1 in /sdks (#17646)
dependabot[bot] [Thu, 12 May 2022 20:44:23 +0000 (13:44 -0700)] 
Bump cloud.google.com/go/pubsub from 1.18.0 to 1.21.1 in /sdks (#17646)

Bumps [cloud.google.com/go/pubsub](https://github.com/googleapis/google-cloud-go) from 1.18.0 to 1.21.1.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.18.0...pubsub/v1.21.1)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/pubsub
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
6 days agoMerge pull request #17584 from [BEAM-14415] Exception handling tests and logging...
Pablo [Thu, 12 May 2022 20:38:55 +0000 (16:38 -0400)] 
Merge pull request #17584 from [BEAM-14415] Exception handling tests and logging for partial failure BQIO

* [BEAM-14415] Exception handling tests and logging for partial failures in BQ IO

* fix DLQ integration test

* fix lint

* fix postcommit

* fix formatter

* Fixing tests and adding test info

* fix skipping tests

6 days agoBump github.com/lib/pq from 1.10.4 to 1.10.5 in /sdks (#17626)
dependabot[bot] [Thu, 12 May 2022 20:06:09 +0000 (13:06 -0700)] 
Bump github.com/lib/pq from 1.10.4 to 1.10.5 in /sdks (#17626)

Bumps [github.com/lib/pq](https://github.com/lib/pq) from 1.10.4 to 1.10.5.
- [Release notes](https://github.com/lib/pq/releases)
- [Commits](https://github.com/lib/pq/compare/v1.10.4...v1.10.5)

---
updated-dependencies:
- dependency-name: github.com/lib/pq
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
6 days agoBump github.com/testcontainers/testcontainers-go in /sdks (#17627)
dependabot[bot] [Thu, 12 May 2022 20:05:11 +0000 (13:05 -0700)] 
Bump github.com/testcontainers/testcontainers-go in /sdks (#17627)

Bumps [github.com/testcontainers/testcontainers-go](https://github.com/testcontainers/testcontainers-go) from 0.12.0 to 0.13.0.
- [Release notes](https://github.com/testcontainers/testcontainers-go/releases)
- [Commits](https://github.com/testcontainers/testcontainers-go/compare/v0.12.0...v0.13.0)

---
updated-dependencies:
- dependency-name: github.com/testcontainers/testcontainers-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
6 days agoBump github.com/google/go-cmp from 0.5.7 to 0.5.8 in /sdks (#17628)
dependabot[bot] [Thu, 12 May 2022 20:04:42 +0000 (13:04 -0700)] 
Bump github.com/google/go-cmp from 0.5.7 to 0.5.8 in /sdks (#17628)

Bumps [github.com/google/go-cmp](https://github.com/google/go-cmp) from 0.5.7 to 0.5.8.
- [Release notes](https://github.com/google/go-cmp/releases)
- [Commits](https://github.com/google/go-cmp/compare/v0.5.7...v0.5.8)

---
updated-dependencies:
- dependency-name: github.com/google/go-cmp
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
6 days agoBetter test assertion. (#17551)
Robert Bradshaw [Thu, 12 May 2022 20:02:14 +0000 (13:02 -0700)] 
Better test assertion. (#17551)

6 days ago[BEAM-14347] Add generic registration feature to CHANGES (#17643)
Danny McCormick [Thu, 12 May 2022 20:00:55 +0000 (16:00 -0400)] 
[BEAM-14347] Add generic registration feature to CHANGES (#17643)

6 days agoMerge pull request #17588 from damccorm/users/damccorm/issueTemplates
Aizhamal Nurmamat kyzy [Thu, 12 May 2022 17:13:12 +0000 (10:13 -0700)] 
Merge pull request #17588 from damccorm/users/damccorm/issueTemplates

[BEAM-14442] Add GitHub issue templates

6 days agoRevert "[BEAM-14429] Force java load test on dataflow runner v2 forceNumInitialBundle...
Yichi Zhang [Thu, 12 May 2022 17:06:16 +0000 (10:06 -0700)] 
Revert "[BEAM-14429] Force java load test on dataflow runner v2 forceNumInitialBundles to 1 (#17576)" (#17609)

This reverts commit 178441dcf0a1e03611c46bdd12ab27907289e310.

6 days agoTrigger go precommits on go mod/sum changes (#17636)
Danny McCormick [Thu, 12 May 2022 14:37:27 +0000 (10:37 -0400)] 
Trigger go precommits on go mod/sum changes (#17636)

6 days agoPare down to fewer templates 17588/head
Danny McCormick [Thu, 12 May 2022 13:30:51 +0000 (09:30 -0400)] 
Pare down to fewer templates

6 days agoTypo & link update (#17633)
Robert Burke [Thu, 12 May 2022 13:17:35 +0000 (06:17 -0700)] 
Typo & link update (#17633)

6 days ago[BEAM-14334] Fix leakage of SparkContext in Spark runner tests to remove forkEvery...
Moritz Mack [Thu, 12 May 2022 13:08:19 +0000 (15:08 +0200)] 
[BEAM-14334] Fix leakage of SparkContext in Spark runner tests to remove forkEvery 1 (#17406)

* [BEAM-14334] Fix leakage of SparkContext in Spark runner tests to remove forkEvery 1 and set provided SparkContext via SparkContextFactory to avoid losing it during a serde roundtrip in TestPipenline.

7 days agofix lint errors
Heejong Lee [Thu, 12 May 2022 05:29:12 +0000 (22:29 -0700)] 
fix lint errors

7 days agoadd urn, type inference for PythonCallableSource
Heejong Lee [Thu, 12 May 2022 03:34:20 +0000 (20:34 -0700)] 
add urn, type inference for PythonCallableSource

7 days ago[BEAM-13695] Add jamm jvm options to Java 11 (#17178)
Kiley Sok [Wed, 11 May 2022 21:48:57 +0000 (14:48 -0700)] 
[BEAM-13695] Add jamm jvm options to Java 11 (#17178)

* Add jamm jvm options to Java 11

* fix test comments

7 days agoCorrectly route go dependency changes to go label (#17632)
Danny McCormick [Wed, 11 May 2022 21:16:50 +0000 (17:16 -0400)] 
Correctly route go dependency changes to go label (#17632)

7 days agoUpdates CHANGES.md to include some recently discovered known issues 17631/head
Chamikara Jayalath [Wed, 11 May 2022 20:33:08 +0000 (13:33 -0700)] 
Updates CHANGES.md to include some recently discovered known issues

7 days ago[BEAM-14347] Add some benchmarks for generic registration (#17613)
Danny McCormick [Wed, 11 May 2022 19:59:21 +0000 (15:59 -0400)] 
[BEAM-14347] Add some benchmarks for generic registration (#17613)

7 days agoCut p4
Danny McCormick [Wed, 11 May 2022 19:57:56 +0000 (15:57 -0400)] 
Cut p4

7 days agoWe don't need outage
Danny McCormick [Wed, 11 May 2022 19:48:56 +0000 (15:48 -0400)] 
We don't need outage

7 days agoAsk for beam version + other dependencies
Danny McCormick [Wed, 11 May 2022 19:45:03 +0000 (15:45 -0400)] 
Ask for beam version + other dependencies

7 days agoRemove python 3.6 postcommit from mass_comment.py (#17630)
Yichi Zhang [Wed, 11 May 2022 19:08:45 +0000 (12:08 -0700)] 
Remove python 3.6 postcommit from mass_comment.py (#17630)

7 days agoMerge pull request #17519: [BEAM-14096] bump junit-quickcheck to 1.0
Kenn Knowles [Wed, 11 May 2022 18:46:04 +0000 (11:46 -0700)] 
Merge pull request #17519: [BEAM-14096] bump junit-quickcheck to 1.0

7 days ago[BEAM-12526] Add Dependabot (#17563)
Danny McCormick [Wed, 11 May 2022 18:09:39 +0000 (14:09 -0400)] 
[BEAM-12526] Add Dependabot (#17563)

* Create dependabot.yml

* Apache header

* See if dependabot will parse now

* Explanatory comment

* Reduce duplication

* Bug + style from feedback

* Change format to make dependabot happy

7 days ago[BEAM-14081] [CdapIO] Add context classes for CDAP plugins (#17104)
Igor Krasavin [Wed, 11 May 2022 16:23:23 +0000 (19:23 +0300)] 
[BEAM-14081] [CdapIO] Add context classes for CDAP plugins (#17104)

* [BEAM-14048] Add ConfigWrapper for building CDAP PluginConfigs

* [BEAM-14048] Fix checkstyle

* [BEAM-14048] Fix warnings

* [BEAM-14048] Fix warnings

* [BEAM-14048] Fix warning

* [BEAM-14048] Fix warning

* [BEAM-14048] Remove unused dependencies

* [BEAM-14048] Add needed dependencies

* [BEAM-14048] Fix spotless

* [BEAM-14048] Fix typo

* [BEAM-14048] Use fori instead of stream

* [BEAM-14048] Suppress warning

* [BEAM-14048] Add used undeclared artifacts

* [BEAM-14048] Change dependencies to test

* Add context.

* Fix dependencies issue

* Add null annotation

* [BEAM-14048] Refactoring

* Add SuppressWarning.

* Fix style.

* Determine dependencies.

* [BEAM-14048] Use CDAP InstantiatorFactory for creating config objects

* [BEAM-14048] Suppress warning

* [BEAM-14081] Refactoring

* Update maven repo

* Update build.gradle

* [BEAM-14081] Refactoring

* [BEAM-14048] Use ServiceNow CDAP dependency from Maven central

* [BEAM-14048] Set macroFields

* [BEAM-14081] Fix javadoc

* [BEAM-14081] Make BatchContextImpl class abstract

Co-authored-by: vitaly.terentyev <vitaly.terentyev@akvelon.com>
Co-authored-by: Alex Kosolapov <alex.kosolapov@gmail.com>
Co-authored-by: Elizaveta Lomteva <elizaveta.lomteva@akvelon.com>
Co-authored-by: Elizaveta Lomteva <57974525+Lizzfox@users.noreply.github.com>
7 days ago[BEAM-11104] Add self-checkpointing to CHANGES.md (#17612)
Jack McCluskey [Wed, 11 May 2022 15:58:53 +0000 (11:58 -0400)] 
[BEAM-11104] Add self-checkpointing to CHANGES.md (#17612)

7 days ago[BEAM-14396] Bump httplib2 upper bound. (#17602)
tvalentyn [Wed, 11 May 2022 13:58:59 +0000 (15:58 +0200)] 
[BEAM-14396] Bump httplib2 upper bound. (#17602)

7 days ago[BEAM-5492] Python Dataflow integration tests should export the pipeline console...
andoni-guzman [Wed, 11 May 2022 13:29:32 +0000 (08:29 -0500)] 
[BEAM-5492] Python Dataflow integration tests should export the pipeline console output to Jenkins Test Result section (#17530)

8 days ago[BEAM-11104] Add self-checkpointing integration test (#17590)
Jack McCluskey [Wed, 11 May 2022 01:34:07 +0000 (21:34 -0400)] 
[BEAM-11104] Add self-checkpointing integration test (#17590)

8 days ago[BEAM-14347] Rename registration package to register (#17603)
Danny McCormick [Tue, 10 May 2022 23:44:29 +0000 (19:44 -0400)] 
[BEAM-14347] Rename registration package to register (#17603)

8 days ago[BEAM-14430] Adding a logical type support for Python callables to Row schema
Heejong Lee [Fri, 6 May 2022 03:26:31 +0000 (20:26 -0700)] 
[BEAM-14430] Adding a logical type support for Python callables to Row schema

8 days agoMerge pull request #17601 from [BEAM-14447] Revert "Merge pull request #17517 from...
Pablo [Tue, 10 May 2022 21:14:19 +0000 (17:14 -0400)] 
Merge pull request #17601 from [BEAM-14447] Revert "Merge pull request #17517 from [BEAM-14383] Improve "FailedRo‚Ķ

[BEAM-14447] Revert "Merge pull request #17517 from [BEAM-14383] Improve "FailedRo‚Ķ

8 days ago[BEAM-14455] Add UUID to sub-schemas for PythonExternalTransform 17605/head
Heejong Lee [Tue, 10 May 2022 21:05:50 +0000 (14:05 -0700)] 
[BEAM-14455] Add UUID to sub-schemas for PythonExternalTransform

8 days ago[BEAM-14229] Fix SyntheticUnboundedSource duplication from checkpoint issue (#17600)
Yichi Zhang [Tue, 10 May 2022 20:39:39 +0000 (13:39 -0700)] 
[BEAM-14229] Fix SyntheticUnboundedSource duplication from checkpoint issue (#17600)

8 days agoMerge pull request #17593 from [BEAM-14423] Add test cases for BigtableIO.BigtableWri...
Pablo [Tue, 10 May 2022 17:28:39 +0000 (13:28 -0400)] 
Merge pull request #17593 from [BEAM-14423] Add test cases for BigtableIO.BigtableWriterFn fails due to writeRecord

[BEAM-14423] Add test cases for BigtableIO.BigtableWriterFn fails due to writeRecord

8 days agoMerge pull request #17592 from [BEAM-14448] add datastore exception handling test
Pablo [Tue, 10 May 2022 17:24:04 +0000 (13:24 -0400)] 
Merge pull request #17592 from [BEAM-14448] add datastore exception handling test

[BEAM-14448] add datastore test

8 days agoRevert "Merge pull request #17517 from [BEAM-14383] Improve "FailedRows" errors retur... 17601/head
Pablo E [Tue, 10 May 2022 16:49:28 +0000 (09:49 -0700)] 
Revert "Merge pull request #17517 from [BEAM-14383] Improve "FailedRows" errors returned by beam.io.WriteToBigQuery"

This reverts commit 358782006e1db86437b3bf61f910db12d654b1e0.

8 days agoMerge pull request #15679 from aromanenko-dev/BEAM-12918-tpcds-jenkins
Etienne Chauchot [Tue, 10 May 2022 13:16:55 +0000 (15:16 +0200)] 
Merge pull request #15679 from aromanenko-dev/BEAM-12918-tpcds-jenkins

Merge pull request #15679: [BEAM-12918] TPC-DS: add Jenkins jobs

9 days ago[BEAM-14347] Add generic registration for accumulators (#17579)
Danny McCormick [Tue, 10 May 2022 00:05:24 +0000 (20:05 -0400)] 
[BEAM-14347] Add generic registration for accumulators (#17579)

9 days ago[BEAM-14169] Add Credentials rotation cron job for clusters (#17383)
Elias Segundo Antonio [Mon, 9 May 2022 19:39:03 +0000 (14:39 -0500)] 
[BEAM-14169] Add Credentials rotation cron job for clusters (#17383)

Add cron groovy job for credentials rotation. As Jenkins already has the `-xe` flag set while executing DSL jobs, if a command returns an error then the job will be stopped and marked as fail. Addded email notifications in case of failure and jobs were separated.

Co-authored-by: Elias Segundo <elias.segundo@luisrazo.local>
9 days ago[BEAM-14423] Add test cases for BigtableIO.BigtableWriterFn fails due to writeRecord 17593/head
Yi Hu [Mon, 9 May 2022 17:11:32 +0000 (13:11 -0400)] 
[BEAM-14423] Add test cases for BigtableIO.BigtableWriterFn fails due to writeRecord

9 days ago[BEAM-14347] Add generic registration functions for iters and emitters (#17574)
Danny McCormick [Mon, 9 May 2022 18:49:02 +0000 (14:49 -0400)] 
[BEAM-14347] Add generic registration functions for iters and emitters (#17574)

9 days ago[BEAM-14448] add datastore test 17592/head
johnjcasey [Mon, 9 May 2022 18:33:28 +0000 (14:33 -0400)] 
[BEAM-14448] add datastore test

9 days agoMerge pull request #17577 from [BEAM-14435] Adding exception handling tests for Spann...
Pablo [Mon, 9 May 2022 18:24:48 +0000 (14:24 -0400)] 
Merge pull request #17577 from [BEAM-14435] Adding exception handling tests for SpannerIO write transform

* Adding exception handling tests for SpannerIO write transform

* fixup

9 days ago[BEAM-14429] Force java load test on dataflow runner v2 forceNumInitialBundles to...
Yichi Zhang [Mon, 9 May 2022 17:38:06 +0000 (10:38 -0700)] 
[BEAM-14429] Force java load test on dataflow runner v2 forceNumInitialBundles to 1 (#17576)

9 days ago[BEAM-14433] Improve Go split error message. (#17575)
Robert Burke [Mon, 9 May 2022 17:23:18 +0000 (10:23 -0700)] 
[BEAM-14433] Improve Go split error message. (#17575)

9 days agoMerge pull request #17555 from [BEAM-14417] Adding exception handling tests for JdbcI...
Pablo [Mon, 9 May 2022 17:13:00 +0000 (13:13 -0400)] 
Merge pull request #17555 from [BEAM-14417] Adding exception handling tests for JdbcIO.Write

* Adding exception handling tests for JdbcIO.Write

* Improving coordination in backoff test

9 days agoMerge pull request #17565 from [BEAM-14413] add Kafka exception test cases
johnjcasey [Mon, 9 May 2022 17:00:24 +0000 (13:00 -0400)] 
Merge pull request #17565 from [BEAM-14413] add Kafka exception test cases

* [BEAM-10529] update test to publish and receive null keys

* BEAM-14413 start work on test cases

* [BEAM-14413] revert py change

* [BEAM-14413] add remaining kafka test cases

* [BEAM-14413] run spotless

* [BEAM-14413] fix race condition in tests

* [BEAM-14413] Rerun spotless

9 days agoMerge pull request #17544 from [BEAM-14415] Exception handling tests for BQIO streami...
Pablo [Mon, 9 May 2022 16:52:15 +0000 (12:52 -0400)] 
Merge pull request #17544 from [BEAM-14415] Exception handling tests for BQIO streaming inserts in Python

* Exception handling tests for BQIO streaming inserts in Python

* Adding non-retriable tests

* Added streaming retries

* Added streaming retries

* fix lint

* fix formatter

9 days ago[BEAM-11104] Allow self-checkpointing SDFs to return without finishing their restrict...
Jack McCluskey [Mon, 9 May 2022 16:26:13 +0000 (12:26 -0400)] 
[BEAM-11104] Allow self-checkpointing SDFs to return without finishing their restriction (#17558)

9 days agoMerge pull request #17559 from [BEAM-14423] Add exception injection tests for Bigtabl...
Yi Hu [Mon, 9 May 2022 16:09:14 +0000 (12:09 -0400)] 
Merge pull request #17559 from [BEAM-14423] Add exception injection tests for BigtableIO read in BigtableIOTest

* [BEAM-14423] Add exception injection tests for BigtableIO read in BigtableIOTest

* Test reading from bigtable when connection start fails

* Test reading from bigtable when split() fails

* Test reading from bigtable when advance() fails

* Also cleaned up deprecated Builder calls

* halfway commit

* Apply good practice for autovalue class

9 days ago [BEAM-14439] [BEAM-12673] Add extra details to PubSub matcher errors #17586
tvalentyn [Mon, 9 May 2022 15:48:55 +0000 (17:48 +0200)] 
 [BEAM-14439] [BEAM-12673] Add extra details to PubSub matcher errors #17586