beam.git
2 days ago[BEAM-3502] Remove usage of proto.Builder.clone() in DatastoreIO (#4449) master
Exprosed [Sat, 20 Jan 2018 01:45:51 +0000 (20:45 -0500)] 
[BEAM-3502] Remove usage of proto.Builder.clone() in DatastoreIO (#4449)

2 days agoMerge pull request #4450 [BEAM-3490] Wrap DistributionData in a DistributionResult
Robert Bradshaw [Sat, 20 Jan 2018 00:14:35 +0000 (16:14 -0800)] 
Merge pull request #4450 [BEAM-3490] Wrap DistributionData in a DistributionResult

2 days agoMerge pull request #4427
Thomas Groh [Fri, 19 Jan 2018 23:54:34 +0000 (15:54 -0800)] 
Merge pull request #4427

Add InboundDataClient

2 days ago[BEAM-3490] Wrap DistributionData in a DistributionResult for FnApiRunner. 4450/head
Robert Bradshaw [Fri, 19 Jan 2018 22:15:07 +0000 (14:15 -0800)] 
[BEAM-3490] Wrap DistributionData in a DistributionResult for FnApiRunner.

2 days agoMerge pull request #4421
Thomas Groh [Fri, 19 Jan 2018 21:33:55 +0000 (13:33 -0800)] 
Merge pull request #4421

Implement a GRPC Provision Service

2 days agoUse platformThreadFactory for default thread pool.
Ilya Figotin [Thu, 18 Jan 2018 23:12:35 +0000 (15:12 -0800)] 
Use platformThreadFactory for default thread pool.

2 days agoMerge pull request #4219: [BEAM-413] Created local annotation for floating point...
Kenn Knowles [Fri, 19 Jan 2018 18:50:13 +0000 (10:50 -0800)] 
Merge pull request #4219: [BEAM-413] Created local annotation for floating point equality warning.

2 days agoMerge pull request #4263: [BEAM-3351] Fix Javadoc formatting issues
Kenn Knowles [Fri, 19 Jan 2018 18:38:45 +0000 (10:38 -0800)] 
Merge pull request #4263: [BEAM-3351] Fix Javadoc formatting issues

2 days agoMerge pull request #4077 from holdenk/BEAM-3142-fix-proto-gen-for-py3
Ahmet Altay [Fri, 19 Jan 2018 17:49:26 +0000 (09:49 -0800)] 
Merge pull request #4077 from holdenk/BEAM-3142-fix-proto-gen-for-py3

[BEAM-3142] Apply futurize on gen_protos

2 days agoUpdate generated version of Fn API code.
Bill Neubauer [Fri, 19 Jan 2018 01:18:27 +0000 (17:18 -0800)] 
Update generated version of Fn API code.

2 days agoMerge pull request #4446 from iemejia/BEAM-3275-kafka1
Jean-Baptiste Onofré [Fri, 19 Jan 2018 15:09:40 +0000 (16:09 +0100)] 
Merge pull request #4446 from iemejia/BEAM-3275-kafka1

[BEAM-3275] Update KafkaIO to support Kafka version 1.0.0

2 days ago[BEAM-3275] Update KafkaIO to support Kafka version 1.0.0 4446/head
Ismaël Mejía [Thu, 18 Jan 2018 22:12:19 +0000 (23:12 +0100)] 
[BEAM-3275] Update KafkaIO to support Kafka version 1.0.0

3 days agoMerge pull request #4428: Add unshaded configurations in the default profile
Kenn Knowles [Fri, 19 Jan 2018 07:25:42 +0000 (23:25 -0800)] 
Merge pull request #4428: Add unshaded configurations in the default profile

3 days agoMerge pull request #4351: [BEAM-3182][Nexmark][SQL] Implement sql query 2
Kenn Knowles [Fri, 19 Jan 2018 07:16:08 +0000 (23:16 -0800)] 
Merge pull request #4351: [BEAM-3182][Nexmark][SQL] Implement sql query 2

3 days ago[BEAM-3496] Add missing dependencies in hadoopInputFormats build.gradle (#4442)
Łukasz Gajowy [Fri, 19 Jan 2018 01:18:22 +0000 (02:18 +0100)] 
[BEAM-3496] Add missing dependencies in hadoopInputFormats build.gradle (#4442)

Add missing dependencies in hadoopInputFormats build.gradle

3 days agoMerge pull request #4423 [BEAM-3486] Fixing Work Progress reporting
Robert Bradshaw [Fri, 19 Jan 2018 01:10:29 +0000 (17:10 -0800)] 
Merge pull request #4423 [BEAM-3486] Fixing Work Progress reporting

3 days agoMerge pull request #4435
Thomas Groh [Thu, 18 Jan 2018 23:49:27 +0000 (15:49 -0800)] 
Merge pull request #4435

Give UnknownPTransformRunnerFactory a real name

3 days agoMerge pull request #4398: [ BEAM-3464] Fix NexmarkLauncher to not cancel jobs otherwi...
Kenn Knowles [Thu, 18 Jan 2018 23:08:05 +0000 (15:08 -0800)] 
Merge pull request #4398: [ BEAM-3464] Fix NexmarkLauncher to not cancel jobs otherwise successful jobs

3 days agoAdd a more complete FnHarness#main method
Thomas Groh [Fri, 12 Jan 2018 00:07:38 +0000 (16:07 -0800)] 
Add a more complete FnHarness#main method

This can be used within tests to configure the channel factory and
stream observer factory used by the harness.

3 days ago[BEAM-2732] Starting refactor of state tracking in Python (#4375)
Pablo [Thu, 18 Jan 2018 21:39:48 +0000 (13:39 -0800)] 
[BEAM-2732] Starting refactor of state tracking in Python (#4375)

Also giving the Python-only state sampler full functionality.

3 days agoGive UnknownPTransformRunnerFactory a real name 4435/head
Thomas Groh [Sat, 16 Dec 2017 02:13:19 +0000 (18:13 -0800)] 
Give UnknownPTransformRunnerFactory a real name

Log the error encountered before throwing an exception to ensure the
failure is explicitly communicated.

3 days ago[BEAM-2863] Migrate away from deprecated methods.
Luke Cwik [Thu, 18 Jan 2018 18:18:42 +0000 (10:18 -0800)] 
[BEAM-2863] Migrate away from deprecated methods.

3 days agoMerge pull request #4384: [BEAM-2795] Use portable constructs in Flink streaming...
Kenn Knowles [Thu, 18 Jan 2018 19:52:34 +0000 (11:52 -0800)] 
Merge pull request #4384: [BEAM-2795] Use portable constructs in Flink streaming translator

3 days agoMerge pull request #4420: [BEAM-3461] Drop redundant beam-YYY-parent for gradle and... samza-runner
Kenn Knowles [Thu, 18 Jan 2018 19:35:48 +0000 (11:35 -0800)] 
Merge pull request #4420: [BEAM-3461] Drop redundant beam-YYY-parent for gradle and use directory names for project names.

3 days ago[BEAM-3217] add HadoopInputFormatIO integration test using DBInputFormat (#4332)
Łukasz Gajowy [Thu, 18 Jan 2018 18:50:19 +0000 (19:50 +0100)] 
[BEAM-3217] add HadoopInputFormatIO integration test using DBInputFormat (#4332)

The kubernetes infrastructure that is needed for the jenkins job
to run is not available for now. We should add it once
the infrastructure is there.

3 days agoMerge pull request #4438 from robertwb/source-counters
Robert Bradshaw [Thu, 18 Jan 2018 17:55:32 +0000 (09:55 -0800)] 
Merge pull request #4438 from robertwb/source-counters

Enable counters from sources on workers.

3 days agoMerge pull request #4431 from robertwb/reshuffle
Robert Bradshaw [Thu, 18 Jan 2018 17:54:44 +0000 (09:54 -0800)] 
Merge pull request #4431 from robertwb/reshuffle

Fix reshuffle test on non-direct runners.

3 days ago[BEAM-2863] Fix translation of side inputs due to missing method.
Luke Cwik [Thu, 18 Jan 2018 17:17:56 +0000 (09:17 -0800)] 
[BEAM-2863] Fix translation of side inputs due to missing method.

3 days agoMerge pull request #4429
Thomas Groh [Thu, 18 Jan 2018 17:21:14 +0000 (09:21 -0800)] 
Merge pull request #4429

Update Dataflow Container Version

4 days ago[BEAM-3456] Enable jenkins and large scale scenario in JDBC (#4392)
Łukasz Gajowy [Thu, 18 Jan 2018 07:20:48 +0000 (08:20 +0100)] 
[BEAM-3456] Enable jenkins and large scale scenario in JDBC (#4392)

[BEAM-3456] Enable jenkins and large scale scenario in JDBC

The kubernetes infrastructure that is needed for the
Jenkins job to run is not available for now.
We should add it once the infrastructure is there.

4 days agolint 4431/head
Robert Bradshaw [Thu, 18 Jan 2018 01:48:24 +0000 (17:48 -0800)] 
lint

4 days agoMerge pull request #4432 from jkff/rm-max-bundle-size
Eugene Kirpichov [Thu, 18 Jan 2018 01:40:41 +0000 (17:40 -0800)] 
Merge pull request #4432 from jkff/rm-max-bundle-size

Removes maxBundleSize option accidentally introduced in unrelated PR #4185

4 days agoEnable counters from sources on workers. 4438/head
Robert Bradshaw [Thu, 18 Jan 2018 01:30:22 +0000 (17:30 -0800)] 
Enable counters from sources on workers.

4 days agoRemoves maxBundleSize option accidentally introduced in unrelated PR #4185 4432/head
Eugene Kirpichov [Wed, 17 Jan 2018 23:56:06 +0000 (15:56 -0800)] 
Removes maxBundleSize option accidentally introduced in unrelated PR #4185

4 days agoFix reshuffle test on non-direct runners.
Robert Bradshaw [Wed, 17 Jan 2018 23:36:37 +0000 (15:36 -0800)] 
Fix reshuffle test on non-direct runners.

Manually specified windowing functions were not being preserved.

4 days agoMerge pull request #4386
Thomas Groh [Wed, 17 Jan 2018 23:35:03 +0000 (15:35 -0800)] 
Merge pull request #4386

[BEAM-3450] Add wire_coder_id to RemoteGrpcPort

4 days agoUse portable constructs in Flink streaming translator 4384/head
Ben Sidhom [Thu, 21 Dec 2017 21:17:31 +0000 (13:17 -0800)] 
Use portable constructs in Flink streaming translator

4 days agoIdentify unconsumed reads by URN
Ben Sidhom [Fri, 12 Jan 2018 19:44:12 +0000 (11:44 -0800)] 
Identify unconsumed reads by URN

4 days agoRevert "Temporarily disable permared Dataflow integration tests" 4429/head
Thomas Groh [Wed, 17 Jan 2018 22:16:10 +0000 (14:16 -0800)] 
Revert "Temporarily disable permared Dataflow integration tests"

This reverts commit 9e536584a09bf538ebb6ef53b2dfd2d8ffe18be6.

4 days agoUpdate Dataflow Container Version
Thomas Groh [Wed, 17 Jan 2018 21:50:02 +0000 (13:50 -0800)] 
Update Dataflow Container Version

4 days agoMerge pull request #4312 from rniemo-g/multiple-row-ranges
Eugene Kirpichov [Wed, 17 Jan 2018 21:23:42 +0000 (13:23 -0800)] 
Merge pull request #4312 from rniemo-g/multiple-row-ranges

[BEAM-3154] Support Multiple KeyRanges when reading from BigTable

4 days agoAdd unshaded configurations in the default profile 4428/head
Kenneth Knowles [Wed, 17 Jan 2018 20:26:34 +0000 (12:26 -0800)] 
Add unshaded configurations in the default profile

4 days ago[Nexmark][SQL] Implement sql query 2 4351/head
Anton Kedin [Tue, 14 Nov 2017 18:26:08 +0000 (10:26 -0800)] 
[Nexmark][SQL] Implement sql query 2

4 days agoAdd InboundDataClient 4427/head
Thomas Groh [Mon, 4 Dec 2017 21:22:03 +0000 (13:22 -0800)] 
Add InboundDataClient

This replaces CompletableFuture<Void> (or any type of Void future) for
use of an active stream that can be completed or cancelled by both
ends.

4 days agoAdd joda-time dependency to java/fn-execution 4386/head
Thomas Groh [Wed, 17 Jan 2018 01:28:23 +0000 (17:28 -0800)] 
Add joda-time dependency to java/fn-execution

4 days agoAdd wire_coder_id to RemoteGrpcPort
Thomas Groh [Tue, 9 Jan 2018 23:51:44 +0000 (15:51 -0800)] 
Add wire_coder_id to RemoteGrpcPort

Use this ID in BeamFnDataReadRunner and BeamFnDataWriteRunner when
populated.

5 days agoMerge pull request #4224 4226/head
Thomas Groh [Wed, 17 Jan 2018 06:13:09 +0000 (22:13 -0800)] 
Merge pull request #4224

Move BeamFnDataGrpcMultiplexer to java-fn-execution

5 days agoMerge pull request #4129: [BEAM-3182][Nexmark][SQL] Implement query 1
Kenn Knowles [Wed, 17 Jan 2018 06:12:16 +0000 (22:12 -0800)] 
Merge pull request #4129: [BEAM-3182][Nexmark][SQL] Implement query 1

5 days agoMerge pull request #4356: [BEAM-2963] Pass portable transform IDs in Dataflow translation
Kenn Knowles [Wed, 17 Jan 2018 05:59:22 +0000 (21:59 -0800)] 
Merge pull request #4356: [BEAM-2963] Pass portable transform IDs in Dataflow translation

5 days agoCorrect getAdditionalInputs for Dataflow single-output ParDo override 4356/head
Kenneth Knowles [Fri, 12 Jan 2018 20:33:07 +0000 (12:33 -0800)] 
Correct getAdditionalInputs for Dataflow single-output ParDo override

5 days agoPass portable transform IDs in Dataflow runner
Kenneth Knowles [Sat, 6 Jan 2018 19:07:54 +0000 (11:07 -0800)] 
Pass portable transform IDs in Dataflow runner

5 days agoMore logging and error checking in ProcessBundleHandler
Kenneth Knowles [Sat, 6 Jan 2018 19:07:29 +0000 (11:07 -0800)] 
More logging and error checking in ProcessBundleHandler

5 days agoFix coders in Java SDK harness
Kenneth Knowles [Sat, 6 Jan 2018 19:07:14 +0000 (11:07 -0800)] 
Fix coders in Java SDK harness

5 days agoMake default ParDo output tag explicit
Kenneth Knowles [Sat, 6 Jan 2018 19:06:32 +0000 (11:06 -0800)] 
Make default ParDo output tag explicit

5 days agoClarify names and visibility of ParDoTranslation helpers
Kenneth Knowles [Sat, 6 Jan 2018 19:05:36 +0000 (11:05 -0800)] 
Clarify names and visibility of ParDoTranslation helpers

5 days agoMove DoFnSignature helpers to DoFnSignatures and rename
Kenneth Knowles [Sat, 6 Jan 2018 19:02:53 +0000 (11:02 -0800)] 
Move DoFnSignature helpers to DoFnSignatures and rename

5 days agoMove DoFnAndMainOutput to top level
Kenneth Knowles [Sat, 6 Jan 2018 18:57:26 +0000 (10:57 -0800)] 
Move DoFnAndMainOutput to top level

5 days agoFixing Work Progress reporting 4423/head
Ankur Goenka [Tue, 16 Jan 2018 23:30:33 +0000 (15:30 -0800)] 
Fixing Work Progress reporting

5 days ago[BEAM-3461] Drop redundant beam-YYY-parent and use directory names for project names. 4420/head
Luke Cwik [Wed, 17 Jan 2018 00:49:40 +0000 (16:49 -0800)] 
[BEAM-3461] Drop redundant beam-YYY-parent and use directory names for project names.

5 days agoMerge pull request #4408 from youngoli/beam3126
Robert Bradshaw [Wed, 17 Jan 2018 01:09:37 +0000 (17:09 -0800)] 
Merge pull request #4408 from youngoli/beam3126

[BEAM-3126] Creating flatten operation in Python SDK Harness

5 days agoImplement a GRPC Provision Service 4421/head
Thomas Groh [Fri, 12 Jan 2018 23:37:59 +0000 (15:37 -0800)] 
Implement a GRPC Provision Service

A Provision Service instance is required to create an environment
container via the Beam container contract.

The current provision service returns a static ProvisionInfo for all
requests.

5 days agoMerge branch 'master' of https://github.com/apache/beam into multiple-row-ranges 4312/head
Ryan Niemocienski [Tue, 16 Jan 2018 23:13:08 +0000 (15:13 -0800)] 
Merge branch 'master' of https://github.com/apache/beam into multiple-row-ranges

5 days agoMerge pull request #4413 from DariuszAniszewski/use-virtualenv-for-perfkit-jobs
Chamikara Jayalath [Tue, 16 Jan 2018 23:02:24 +0000 (15:02 -0800)] 
Merge pull request #4413 from DariuszAniszewski/use-virtualenv-for-perfkit-jobs

[BEAM-3483] use VirtualEnv for PerfKit jobs

5 days agoUse multiple key ranges in BigtableServiceImpl
Ryan Niemocienski [Thu, 21 Dec 2017 20:14:37 +0000 (12:14 -0800)] 
Use multiple key ranges in BigtableServiceImpl

5 days agoAdd tests for supporting multiple key ranges.
Ryan Niemocienski [Thu, 21 Dec 2017 18:54:32 +0000 (10:54 -0800)] 
Add tests for supporting multiple key ranges.

- Add tests for BigTableIO

5 days agoUpdate BigTableSource to use multiple key ranges
Ryan Niemocienski [Wed, 20 Dec 2017 23:25:35 +0000 (15:25 -0800)] 
Update BigTableSource to use multiple key ranges

- Exposes multiple key range functionality to the user
- Modifies BigTableSource to split and estimate size with multiple
ranges.

5 days agoMerge pull request #4185 from rmannibucau/fb/BEAM-3243_better-error-message-for-anony...
Eugene Kirpichov [Tue, 16 Jan 2018 21:17:31 +0000 (13:17 -0800)] 
Merge pull request #4185 from rmannibucau/fb/BEAM-3243_better-error-message-for-anonymous-name-conflicts

BEAM-3243 better error message when there are conflicting anonymous names

5 days ago[BEAM-3126] Fixes for flatten runner.
Daniel Oliveira [Sat, 13 Jan 2018 01:09:28 +0000 (17:09 -0800)] 
[BEAM-3126] Fixes for flatten runner.

Fixed some comments and changed the FlattenRunner to work with only
one output.

5 days ago[BEAM-3125] Fixing little nitpicks and typos.
Daniel Oliveira [Fri, 5 Jan 2018 19:03:23 +0000 (11:03 -0800)] 
[BEAM-3125] Fixing little nitpicks and typos.

5 days ago[BEAM-3125] Creating flatten runner
Daniel Oliveira [Fri, 5 Jan 2018 02:12:01 +0000 (18:12 -0800)] 
[BEAM-3125] Creating flatten runner

Made a runner for flatten transformations and added a unit test for
it. Also fixed a typo in FnApiDoFnRunnerTest.

5 days ago[BEAM-3125] Creating flatten runner, unfinished.
Daniel Oliveira [Sat, 23 Dec 2017 00:50:29 +0000 (16:50 -0800)] 
[BEAM-3125] Creating flatten runner, unfinished.

Need to commit it so I can work on this from another machine.

5 days agoUpdate GRPC Multiplexer and Associated to Java8 4224/head
Thomas Groh [Thu, 11 Jan 2018 22:35:27 +0000 (14:35 -0800)] 
Update GRPC Multiplexer and Associated to Java8

5 days agoMerge pull request #4379 from jbonofre/BEAM-3438
Ismaël Mejía [Tue, 16 Jan 2018 16:43:58 +0000 (17:43 +0100)] 
Merge pull request #4379 from jbonofre/BEAM-3438

[BEAM-3438] Disable IT on KinesisIO

5 days ago[BEAM-3438] Disable it on Kinesis 4379/head
Jean-Baptiste Onofré [Tue, 16 Jan 2018 10:40:50 +0000 (11:40 +0100)] 
[BEAM-3438] Disable it on Kinesis

6 days agoMerge pull request #4402: [BEAM-3240] Fix quickstart archetypes to include expected...
Thomas Weise [Tue, 16 Jan 2018 06:53:59 +0000 (22:53 -0800)] 
Merge pull request #4402: [BEAM-3240] Fix quickstart archetypes to include expected Hadoop dependencies under Apex profile

6 days agoMerge pull request #4415 from RyanSkraba/BEAM-3454-stop-unbounded-srcs
Ismaël Mejía [Mon, 15 Jan 2018 16:05:10 +0000 (17:05 +0100)] 
Merge pull request #4415 from RyanSkraba/BEAM-3454-stop-unbounded-srcs

[BEAM-3454] Use multiple stopping conditions on JmsIO, AmqpIO, MqttIO.

6 days agouse VirtualEnv for PerfKit jobs 4413/head
Dariusz Aniszewski [Mon, 15 Jan 2018 10:02:35 +0000 (11:02 +0100)] 
use VirtualEnv for PerfKit jobs

6 days ago[BEAM-3454] Use multiple stopping conditions on JmsIO, AmqpIO, MqttIO. 4415/head
Ryan Skraba [Mon, 15 Jan 2018 14:41:04 +0000 (15:41 +0100)] 
[BEAM-3454] Use multiple stopping conditions on JmsIO, AmqpIO, MqttIO.

6 days agoMerge pull request #4414 from iemejia/master
Ismaël Mejía [Mon, 15 Jan 2018 14:29:41 +0000 (15:29 +0100)] 
Merge pull request #4414 from iemejia/master

Add missing protobuf-java dependency on runners/java-fn-execution module

6 days agoAdd missing protobuf-java dependency on runners/java-fn-execution module 4414/head
Ismaël Mejía [Mon, 15 Jan 2018 13:03:13 +0000 (14:03 +0100)] 
Add missing protobuf-java dependency on runners/java-fn-execution module

7 days agoadressing comments 4185/head
Romain Manni-Bucau [Sun, 14 Jan 2018 15:29:02 +0000 (16:29 +0100)] 
adressing comments

8 days agoMerge pull request #4400 from iemejia/java8-update
Ismaël Mejía [Sat, 13 Jan 2018 21:37:29 +0000 (22:37 +0100)] 
Merge pull request #4400 from iemejia/java8-update

[BEAM-3467] Remove Java 7 from the docker development images

8 days agoMerge pull request #4409: [BEAM-3397] Temporarily disable permared Dataflow integrati...
Kenn Knowles [Sat, 13 Jan 2018 19:34:02 +0000 (11:34 -0800)] 
Merge pull request #4409: [BEAM-3397] Temporarily disable permared Dataflow integration tests

8 days agoMerge pull request #4411: Fix unused dependency error on Flink runner
Kenn Knowles [Sat, 13 Jan 2018 19:33:25 +0000 (11:33 -0800)] 
Merge pull request #4411: Fix unused dependency error on Flink runner

8 days agoFix unused dependency error on Flink runner 4411/head
Ismaël Mejía [Sat, 13 Jan 2018 13:33:14 +0000 (14:33 +0100)] 
Fix unused dependency error on Flink runner

8 days agoMerge pull request #4410 from lukecwik/mqtt
Ismaël Mejía [Sat, 13 Jan 2018 13:29:56 +0000 (14:29 +0100)] 
Merge pull request #4410 from lukecwik/mqtt

[BEAM-3478] Fix checkstyle issue in runners/flink

9 days agoMerge pull request #4235 from rmannibucau/fb/BEAM-682_fixing-bytebuddy-classloader...
Jean-Baptiste Onofré [Sat, 13 Jan 2018 06:58:23 +0000 (07:58 +0100)] 
Merge pull request #4235 from rmannibucau/fb/BEAM-682_fixing-bytebuddy-classloader-usage

[BEAM-682] try to define generated classes in the correct classloader instead of the beam classloader by default

9 days ago[BEAM-3478] Fix checkstyle issue in runners/flink 4410/head
Luke Cwik [Sat, 13 Jan 2018 06:14:20 +0000 (22:14 -0800)] 
[BEAM-3478] Fix checkstyle issue in runners/flink

9 days ago[BEAM-3336] Fix thread safety issues of MqttIOTest (#4406)
Lukasz Cwik [Sat, 13 Jan 2018 06:14:01 +0000 (22:14 -0800)] 
[BEAM-3336] Fix thread safety issues of MqttIOTest (#4406)

* [BEAM-3336] Fix thread safety issues of MqttIOTest.

Ran the tests on my desktop 50 times with zero flakes.

* fixup! Adjust test timeouts

9 days ago[BEAM-3436] Fix RetryHttpRequestInitializerTest to use fake clock/sleeper instead...
Lukasz Cwik [Sat, 13 Jan 2018 05:12:20 +0000 (21:12 -0800)] 
[BEAM-3436] Fix RetryHttpRequestInitializerTest to use fake clock/sleeper instead of actually waiting. (#4407)

* [BEAM-3436] Fix RetryHttpRequestInitializerTest to use fake clock/sleeper instead of actually waiting.

* fixup!

9 days agoMerge pull request #4343: [BEAM-2795] Use portable constructs in Flink batch translator
Kenn Knowles [Sat, 13 Jan 2018 01:31:28 +0000 (17:31 -0800)] 
Merge pull request #4343: [BEAM-2795] Use portable constructs in Flink batch translator

9 days agoTemporarily disable permared Dataflow integration tests 4409/head
Kenneth Knowles [Sat, 13 Jan 2018 01:24:22 +0000 (17:24 -0800)] 
Temporarily disable permared Dataflow integration tests

9 days agoMerge pull request #4405
Thomas Groh [Sat, 13 Jan 2018 00:33:22 +0000 (16:33 -0800)] 
Merge pull request #4405

Handle a populated Error field in FnApiControlClient

9 days ago[BEAM-3126] Creating flatten operation for bundle processor. 4408/head
Daniel Oliveira [Sat, 13 Jan 2018 00:20:29 +0000 (16:20 -0800)] 
[BEAM-3126] Creating flatten operation for bundle processor.

Added the flatten URN to the URNs registered with the
BeamTransformFactory.

9 days agoMerge pull request #4390 from chamikaramj/bq_read_non_existing_table
Chamikara Jayalath [Sat, 13 Jan 2018 00:04:01 +0000 (16:04 -0800)] 
Merge pull request #4390 from chamikaramj/bq_read_non_existing_table

[BEAM-3452] Updates the error message when trying to execute an export job on a non-existing BQ table.

9 days ago[BEAM-2863] Add the ability to length prefix unknown coders using the portable repres...
Luke Cwik [Wed, 10 Jan 2018 01:28:46 +0000 (17:28 -0800)] 
[BEAM-2863] Add the ability to length prefix unknown coders using the portable representation allowing a Runner to not need to know about all coder representations.

This is towards supporting the side inputs over the portability framework.

9 days agoUse portable constructs in Flink batch translator 4343/head
Ben Sidhom [Fri, 12 Jan 2018 20:28:39 +0000 (12:28 -0800)] 
Use portable constructs in Flink batch translator

This was tested by round-tripping batch pipelines to and from protobuf
form. It works with both real Java pipelines and rehydrated pipelines.

References and downcasts to specific transform subclasses are replaced
with generic `PTransform`s. Transform metadata is now accessed through
the translation utilities under
`org.apache.beam.runners.core.construction`.

The `ParDo` union coder is picky about ordering. It appears that coders
must appear at the same indexes as their respective output collection
tags. This ordering is now preserved.

9 days agoExpose side inputs in CombineTranslation and refactor portable translation
Ben Sidhom [Fri, 12 Jan 2018 20:22:06 +0000 (12:22 -0800)] 
Expose side inputs in CombineTranslation and refactor portable translation

`CombineTranslation` uses a new side input extractor modeled after
`ParDoTranslation#getSideInputs`.

The `RawCombine` rehydrated transform exposes side inputs via
`getAdditionalInputs`. Side inputs were not previously exposed as
"additional" inputs, so portable translators could not properly extract
the main output collection when side inputs were used.

`ParDoTranslation.viewFromProto` was used all over this package for
general view translations. This method has been moved into a new
`PCollectionViewTranslation` class.

9 days agoMerge pull request #4341: [BEAM-3354] Fixed a bug that prevented processing time...
Kenn Knowles [Fri, 12 Jan 2018 20:35:56 +0000 (12:35 -0800)] 
Merge pull request #4341: [BEAM-3354] Fixed a bug that prevented processing time timers to be reset

9 days agoEnable Java8 for fn execution modules
Thomas Groh [Thu, 11 Jan 2018 19:29:04 +0000 (11:29 -0800)] 
Enable Java8 for fn execution modules