incubator-airflow.git
10 hours ago[AIRFLOW-3506] use match_phrase to query log_id in elasticsearch (#4342) master
Ping Zhang [Tue, 18 Dec 2018 22:51:10 +0000 (14:51 -0800)] 
[AIRFLOW-3506] use match_phrase to query log_id in elasticsearch (#4342)

38 hours ago[AIRFLOW-XXX] Added LeMans Corporation as user of Airflow in README (#4334)
Adam Whitlock [Mon, 17 Dec 2018 19:26:21 +0000 (13:26 -0600)] 
[AIRFLOW-XXX] Added LeMans Corporation as user of Airflow in README (#4334)

39 hours ago[AIRFLOW-XXX] Add missing remote logging field (#4333)
Felix [Mon, 17 Dec 2018 18:05:21 +0000 (19:05 +0100)] 
[AIRFLOW-XXX] Add missing remote logging field (#4333)

2 days ago[AIRFLOW-2770] Read `dags_in_image` config value as a boolean (#4319)
Kevin Pullin [Mon, 17 Dec 2018 07:05:26 +0000 (23:05 -0800)] 
[AIRFLOW-2770] Read `dags_in_image` config value as a boolean (#4319)

* Read `dags_in_image` config value as a boolean

This PR is a minor fix for #3683

The dags_in_image config value is read as a string. However, the existing code expects this to be a boolean.

For example, in worker_configuration.py there is the statement: if not self.kube_config.dags_in_image:

Since the value is a non-empty string ('False') and not a boolean, this evaluates to true (since non-empty strings are truthy)
and skips the logic to add the dags_volume_claim volume mount.

This results in the CI tests failing because the dag volume is missing in the k8s pod definition.

This PR reads the dags_in_image using the conf.getboolean to fix this error.

Rebased on 457ad83e4eb02b7348e5ce00292ca9bd27032651, before the previous
dags_in_image commit was reverted.

* Revert "Revert  [AIRFLOW-2770] [AIRFLOW-3505] (#4318)"

This reverts commit 77c368fd228fe5edfdb3304ed4cb000a50667010.

2 days ago[AIRFLOW-XXX] Add Société générale to company list (#4330)
Med [Sun, 16 Dec 2018 23:45:12 +0000 (00:45 +0100)] 
[AIRFLOW-XXX] Add Société générale to company list (#4330)

3 days ago[AIRFLOW-3500] Make task duration display user friendly (#4304)
Ofer Zelig [Sun, 16 Dec 2018 03:20:17 +0000 (14:20 +1100)] 
[AIRFLOW-3500] Make task duration display user friendly (#4304)

3 days ago[AIRFLOW-3447] Add 2 options for ts_nodash Macro (#4323)
Kaxil Naik [Sat, 15 Dec 2018 23:13:36 +0000 (23:13 +0000)] 
[AIRFLOW-3447] Add 2 options for ts_nodash Macro (#4323)

3 days ago[AIRFLOW-1552] Airflow Filter_by_owner not working with password_auth (#4276)
thomasbrockmeier [Sat, 15 Dec 2018 15:27:10 +0000 (16:27 +0100)] 
[AIRFLOW-1552] Airflow Filter_by_owner not working with password_auth (#4276)

Local users were always a superuser, this adds a column to the DB (and defaults to false,
which is going to cause a bit of an upgrade pain for people, but defaulting to not being an
admin is the only secure default.)

3 days ago[AIRFLOW-3518] Performance fixes for topological_sort of Tasks (#4322)
Niels Zeilemaker [Sat, 15 Dec 2018 15:11:26 +0000 (16:11 +0100)] 
[AIRFLOW-3518] Performance fixes for topological_sort of Tasks (#4322)

For larger DAGs topological_sort was found to be very inefficient. Made
some small changes to the code to improve the data structures used in the
method.

3 days ago[AIRFLOW-3521] Fetch more than 50 items in `airflow-jira compare` script (#4300)
Ash Berlin-Taylor [Sat, 15 Dec 2018 11:50:22 +0000 (11:50 +0000)] 
[AIRFLOW-3521] Fetch more than 50 items in `airflow-jira compare` script (#4300)

4 days ago[AIRFLOW-3444] Explicitly set transfer operator description. (#4279)
Joshua Carp [Fri, 14 Dec 2018 12:23:23 +0000 (07:23 -0500)] 
[AIRFLOW-3444] Explicitly set transfer operator description. (#4279)

5 days ago[AIRFLOW-3452] removed an unused/dangerous display-none (#4295)
Marcus [Fri, 14 Dec 2018 07:19:22 +0000 (23:19 -0800)] 
[AIRFLOW-3452] removed an unused/dangerous display-none (#4295)

* removed an unused display-none that is currently overriden but could resurface as a bug.

* remove the other display none in /www

5 days agoRevert [AIRFLOW-2770] [AIRFLOW-3505] (#4318)
Tao Feng [Thu, 13 Dec 2018 18:21:39 +0000 (10:21 -0800)] 
Revert  [AIRFLOW-2770] [AIRFLOW-3505] (#4318)

* Revert "[AIRFLOW-3505] replace 'dags_in_docker' with 'dags_in_image' (#4311)"

This reverts commit 457ad83e4eb02b7348e5ce00292ca9bd27032651.

* Revert "[AIRFLOW-2770] kubernetes: add support for dag folder in the docker image (#3683)"

This reverts commit e9a09d408e4cd1bda1d6e8b7670f08beab37de8a.

6 days ago[AIRFLOW-3505] replace 'dags_in_docker' with 'dags_in_image' (#4311)
Daniel Imberman [Thu, 13 Dec 2018 05:03:41 +0000 (21:03 -0800)] 
[AIRFLOW-3505] replace 'dags_in_docker' with 'dags_in_image' (#4311)

As kubernetes is moving away from docker to OCI, it will be more correct to use the
'dags_in_image' name to be more container system agnostic

6 days ago[AIRFLOW-3411] Add OpenFaaS hook (#4267)
tal181 [Thu, 13 Dec 2018 01:23:47 +0000 (03:23 +0200)] 
[AIRFLOW-3411]  Add OpenFaaS hook (#4267)

6 days ago[AIRFLOW-3310] Google Cloud Spanner deploy / delete operators (#4286)
Szymon Przedwojski [Thu, 13 Dec 2018 01:15:43 +0000 (02:15 +0100)] 
[AIRFLOW-3310] Google Cloud Spanner deploy / delete operators (#4286)

6 days ago[AIRFLOW-3502] Add celery config option for setting "pool" (#4308)
Gabriel Silk [Wed, 12 Dec 2018 17:59:03 +0000 (09:59 -0800)] 
[AIRFLOW-3502] Add celery config option for setting "pool" (#4308)

6 days ago[AIRFLOW-2770] kubernetes: add support for dag folder in the docker image (#3683)
Rurui [Wed, 12 Dec 2018 17:43:58 +0000 (01:43 +0800)] 
[AIRFLOW-2770] kubernetes: add support for dag folder in the docker image (#3683)

6 days ago[AIRFLOW-XXX] add ARMEDANGELS to the list of customers (#4310)
Matthias Wirtz [Wed, 12 Dec 2018 17:31:02 +0000 (18:31 +0100)] 
[AIRFLOW-XXX] add ARMEDANGELS to the list of customers (#4310)

7 days ago[AIRFLOW-3397] Fix integrety error in rbac AirflowSecurityManager (#4305)
BrechtDeVlieger [Tue, 11 Dec 2018 18:20:14 +0000 (19:20 +0100)] 
[AIRFLOW-3397] Fix integrety error in rbac AirflowSecurityManager (#4305)

This was caused by the variable `role` being shadowed in a loop statement.

9 days ago[AIRFLOW-987] pass kerberos cli args keytab and principal to kerberos.run() (#4238)
Iuliia Volkova [Sun, 9 Dec 2018 22:48:39 +0000 (01:48 +0300)] 
[AIRFLOW-987] pass kerberos cli args keytab and principal to kerberos.run() (#4238)

9 days ago[AIRFLOW-3438] Fix default values in BigQuery Hook & BigQueryOperator (#4274)
Kaxil Naik [Sun, 9 Dec 2018 22:29:11 +0000 (22:29 +0000)] 
[AIRFLOW-3438] Fix default values in BigQuery Hook & BigQueryOperator (#4274)

10 days ago[AIRFLOW-XXX] Fix Minor issues with Azure Cosmos Operator (#4289)
Kaxil Naik [Sat, 8 Dec 2018 22:31:39 +0000 (22:31 +0000)] 
[AIRFLOW-XXX] Fix Minor issues with Azure Cosmos Operator (#4289)

- Fixed Documentation in integration.rst
- Fixed Incorrect type in docstring of `AzureCosmosInsertDocumentOperator`
- Added the Hook, Sensor and Operator in code.rst
- Updated the name of example DAG and its filename to follow the convention

11 days ago[AIRFLOW-3479] Keeps records in Log Table when DAG is deleted (#4287)
Xiaodong [Sat, 8 Dec 2018 01:14:43 +0000 (09:14 +0800)] 
[AIRFLOW-3479] Keeps records in Log Table when DAG is deleted (#4287)

Users will use either API or web UI to delete DAG (after DAG file is
removed):

- Using API: provide one boolean parameter to let users
             decide if they want to keep records in Log table
             when they delete a DAG.
             Default value it True (to keep records in Log table).
- From UI: will keep records in the Log table when delete records for a
           specific DAG ID (pop-up message is updated accordingly).

11 days ago[AIRFLOW-3484] Fix Over-logging in the k8s executor (#4296)
Daniel Imberman [Fri, 7 Dec 2018 23:39:47 +0000 (15:39 -0800)] 
[AIRFLOW-3484] Fix Over-logging in the k8s executor (#4296)

There are two log lines in the k8sexecutor that can cause schedulers to crash
due to too many logs.

12 days ago[AIRFLOW-2524] Add SageMaker doc to AWS integration section (#4278)
yangaws [Thu, 6 Dec 2018 19:51:11 +0000 (11:51 -0800)] 
[AIRFLOW-2524] Add SageMaker doc to AWS integration section (#4278)

12 days ago[AIRFLOW-3406] Implement an Azure CosmosDB operator (#4265)
Tom Miller [Thu, 6 Dec 2018 18:18:44 +0000 (10:18 -0800)] 
[AIRFLOW-3406] Implement an Azure CosmosDB operator (#4265)

Add an operator and hook to manipulate and use Azure
CosmosDB documents, including creation, deletion, and
updating documents and collections.

Includes sensor to detect documents being added to a
collection.

12 days ago[AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template (#4282)
Kaxil Naik [Thu, 6 Dec 2018 09:58:59 +0000 (09:58 +0000)] 
[AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template (#4282)

* [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template

* Update PULL_REQUEST_TEMPLATE.md

13 days ago[AIRFLOW-3322] Update QuboleHook to fetch args dynamically from qds_sdk (#4165)
Joy Lal Chattaraj [Thu, 6 Dec 2018 08:20:07 +0000 (13:50 +0530)] 
[AIRFLOW-3322] Update QuboleHook to fetch args dynamically from qds_sdk (#4165)

13 days ago[AIRFLOW-3408] Remove outdated info from Systemd Instructions (#4269)
Kaxil Naik [Wed, 5 Dec 2018 20:50:16 +0000 (20:50 +0000)] 
[AIRFLOW-3408] Remove outdated info from Systemd Instructions (#4269)

13 days ago[AIRFLOW-XXX] GCP operators documentation clarifications (#4273)
Jarek Potiuk [Wed, 5 Dec 2018 20:35:29 +0000 (21:35 +0100)] 
[AIRFLOW-XXX] GCP operators documentation clarifications (#4273)

13 days ago[AIRFLOW-2440] Google Cloud SQL import/export operator (#4251)
Szymon Przedwojski [Wed, 5 Dec 2018 20:33:00 +0000 (21:33 +0100)] 
[AIRFLOW-2440] Google Cloud SQL import/export operator (#4251)

13 days ago[AIRFLOW-XXX] Add Kubernetes Dependency in Extra Packages Doc (#4281)
Gabriel Nicolas Avellaneda [Wed, 5 Dec 2018 19:55:38 +0000 (17:55 -0200)] 
[AIRFLOW-XXX] Add Kubernetes Dependency in Extra Packages Doc (#4281)

13 days ago[AIRFLOW-XXX] Update kubernetes.rst (#4280)
SUNNY [Wed, 5 Dec 2018 18:17:55 +0000 (23:47 +0530)] 
[AIRFLOW-XXX] Update kubernetes.rst (#4280)

import modules to complete the example set.

2 weeks ago[AIRFLOW-XXX] Add Get Simpl to Companies (#4272) 4231/head
Shekhar Singh [Mon, 3 Dec 2018 15:45:21 +0000 (21:15 +0530)] 
[AIRFLOW-XXX] Add Get Simpl to Companies (#4272)

2 weeks ago[AIRFLOW-3434] Allows creating intermediate dirs in SFTPOperator (#4270)
Kaxil Naik [Mon, 3 Dec 2018 12:39:46 +0000 (12:39 +0000)] 
[AIRFLOW-3434] Allows creating intermediate dirs in SFTPOperator (#4270)

2 weeks ago[AIRFLOW-3431] Document how to report security vulnerabilities. (#4262)
Ash Berlin-Taylor [Mon, 3 Dec 2018 10:01:53 +0000 (10:01 +0000)] 
[AIRFLOW-3431] Document how to report security vulnerabilities. (#4262)

Wording based on Kafka's

[ci-skip]

2 weeks ago[AIRFLOW-3367] Run celery integration test with redis broker. (#4207)
Joshua Carp [Mon, 3 Dec 2018 09:19:25 +0000 (04:19 -0500)] 
[AIRFLOW-3367] Run celery integration test with redis broker. (#4207)

2 weeks ago[AIRFLOW-XXX] Fix display of SageMaker operators/hook docs (#4263)
Ash Berlin-Taylor [Sun, 2 Dec 2018 20:51:30 +0000 (20:51 +0000)] 
[AIRFLOW-XXX] Fix display of SageMaker operators/hook docs (#4263)

2 weeks ago[AIRFLOW-3414] Fix reload_module in DagFileProcessorAgent (#4253)
Kevin Yang [Sun, 2 Dec 2018 17:45:08 +0000 (09:45 -0800)] 
[AIRFLOW-3414] Fix reload_module in DagFileProcessorAgent (#4253)

2 weeks ago[AIRFLOW-3432] Add test for feature "Delete DAG in UI" (#4266)
Xiaodong [Sun, 2 Dec 2018 16:54:23 +0000 (00:54 +0800)] 
[AIRFLOW-3432] Add test for feature "Delete DAG in UI" (#4266)

Related Commits:
1. [AIRFLOW-2657](PR #3531)
2. [AIRFLOW-3233](PR #4069)

Added for both www/ and www_rbac

2 weeks ago[AIRFLOW-3416] Fixes Python 3 compatibility with CloudSqlQueryOperator (#4254)
Jarek Potiuk [Sun, 2 Dec 2018 11:09:29 +0000 (12:09 +0100)] 
[AIRFLOW-3416] Fixes Python 3 compatibility with CloudSqlQueryOperator (#4254)

Added several missing decodes on reading output from running
subprocess (cloud_sql_proxy)

2 weeks ago[AIRFLOW-XXX] Add missing GCP operators to Docs (#4260)
Kaxil Naik [Sun, 2 Dec 2018 11:08:26 +0000 (11:08 +0000)] 
[AIRFLOW-XXX] Add missing GCP operators to Docs (#4260)

2 weeks ago[AIRFLOW-3426] Bugfix / Correct Python Version Documentation Reference (#4259)
Bryant Biggs [Sun, 2 Dec 2018 05:58:50 +0000 (00:58 -0500)] 
[AIRFLOW-3426] Bugfix / Correct Python Version Documentation Reference (#4259)

- Change all Python 3.4 references to 3.5 as 3.4 to reflect the currently supported version within the CI test suite (2.7 and 3.5)

2 weeks ago[AIRFLOW-3396] Make sql param as required in BigQueryOperator (#4224)
Kaxil Naik [Sat, 1 Dec 2018 21:35:15 +0000 (21:35 +0000)] 
[AIRFLOW-3396] Make sql param as required in BigQueryOperator (#4224)

2 weeks ago[AIRFLOW-3403] Add AWS Athena Sensor (#4244)
tal181 [Sat, 1 Dec 2018 21:14:20 +0000 (23:14 +0200)] 
[AIRFLOW-3403] Add AWS Athena Sensor (#4244)

2 weeks ago[AIRFLOW-3425] Fix setting default scope in hook (#4261)
Alex Sokolov [Sat, 1 Dec 2018 18:28:30 +0000 (20:28 +0200)] 
[AIRFLOW-3425] Fix setting default scope in hook (#4261)

2 weeks ago[AIRFLOW-3239] Fix/refine tests for api/common/experimental/ (#4255)
Xiaodong [Thu, 29 Nov 2018 15:21:06 +0000 (23:21 +0800)] 
[AIRFLOW-3239] Fix/refine tests for api/common/experimental/ (#4255)

Follow-up on [AIRFLOW-3239]
Related PRs: #4074, #4131

1. Fix (test_)trigger_dag.py

2. Fix (test_)mark_tasks.py
  2-1. properly name the file
  2-2. Correct the name of sample DAG
  2-3. Correct the range of sample execution_dates
       (earlier one conflict with the start_date of the sample DAG)
  2-4. Skip for test running on MySQL
       Seems something is wrong with
       airflow.api.common.experimental.mark_tasks.set_state,
       Corresponding test case works on Postgres & SQLite,
       but fails when on MySQL ("(1062, "Duplicate entry '110' for key 'PRIMARY'")").

       A TODO note is added to remind us fix it for MySQL later.

3. Remove unnecessary lines in test_pool.py

2 weeks ago[AIRFLOW-XXX] Add Poshmark to Companies (#4252)
Adityan [Thu, 29 Nov 2018 04:06:59 +0000 (20:06 -0800)] 
[AIRFLOW-XXX] Add Poshmark to Companies (#4252)

* Add Poshmark to Companies

2 weeks ago[AIRFLOW-2761] Parallelize enqueue in celery executor (#4234)
Kevin Yang [Wed, 28 Nov 2018 22:23:44 +0000 (14:23 -0800)] 
[AIRFLOW-2761] Parallelize enqueue in celery executor (#4234)

2 weeks ago[AIRFLOW-XXX] Add VeeR VR to org list in README (#4250)
pishilong [Wed, 28 Nov 2018 17:52:09 +0000 (01:52 +0800)] 
[AIRFLOW-XXX] Add VeeR VR to org list in README (#4250)

* add VeeR VR to org list in README

* 1. [VeeR VR](https://veer.tv) [[@pishilong](https://github.com/pishilong)]

2 weeks ago[AIRFLOW-3410] Add feature to allow Host Key Change for SSH Op (#4249)
Kaxil Naik [Wed, 28 Nov 2018 16:52:15 +0000 (16:52 +0000)] 
[AIRFLOW-3410] Add feature to allow Host Key Change for SSH Op (#4249)

2 weeks ago[AIRFLOW-3384] Allow higher versions of Sqlalchemy and Jinja2 (#4227)
Jose Luis Ricon [Wed, 28 Nov 2018 09:57:23 +0000 (09:57 +0000)] 
[AIRFLOW-3384] Allow higher versions of Sqlalchemy and Jinja2 (#4227)

* [AIRFLOW-3384] Allow higher versions of SQLAlchemy and Jinja2

3 weeks ago[AIRFLOW-XXX] Replace airflow with apache-airflow (#4246)
Fokko Driesprong [Tue, 27 Nov 2018 19:53:58 +0000 (20:53 +0100)] 
[AIRFLOW-XXX] Replace airflow with apache-airflow (#4246)

3 weeks ago[AIRFLOW-XXX] Remove unnecessary "# noqa" in airflow/bin/cli.py (#4223)
Xiaodong [Tue, 27 Nov 2018 11:03:11 +0000 (19:03 +0800)] 
[AIRFLOW-XXX] Remove unnecessary "# noqa" in airflow/bin/cli.py (#4223)

3 weeks ago[AIRFLOW-XXX] Add TEK to list of companies (#4240)
Christian Westerlund [Tue, 27 Nov 2018 07:20:45 +0000 (09:20 +0200)] 
[AIRFLOW-XXX] Add TEK to list of companies (#4240)

3 weeks ago[AIRFLOW-XXX] Correct typos in UPDATING.md (#4242)
Daniel Lamblin [Tue, 27 Nov 2018 06:14:24 +0000 (15:14 +0900)] 
[AIRFLOW-XXX] Correct typos in UPDATING.md (#4242)

Started with "habe", "serever" and "certificiate" needing to be:
  "have", "server", and "certificate".
Ran a check, ignoring British and US accepted spellings.
Kept jargon. EG admin, aync, auth, backend, config, dag, s3, utils, etc.
Took exception to: "num of dag run" meaning "number of dag runs",
  "upness" is normally for quarks,
  "url" being lower-case, and
  sftp example having an excess file ending.
Python documentation writes "builtin" hyphenated, cases "PYTHONPATH".
Gave up on mixed use of "dag" and "DAG" as well as long line lengths.

3 weeks ago[AIRFLOW-3392] Add index on dag_id in sla_miss table (#4235)
Kevin Yang [Tue, 27 Nov 2018 05:49:31 +0000 (21:49 -0800)] 
[AIRFLOW-3392] Add index on dag_id in sla_miss table (#4235)

The select queries on sla_miss table produce a great % of DB traffic and
thus made the DB CPU usage unnecessarily high. It would be a low hanging
fruit to add an index and reduce the load.

3 weeks ago[AIRFLOW-3395] added the REST API endpoints to the doc (#4236)
Iuliia Volkova [Mon, 26 Nov 2018 09:58:33 +0000 (12:58 +0300)] 
[AIRFLOW-3395] added the REST API endpoints to the doc (#4236)

3 weeks ago[AIRFLOW-XXX] Remove quotes from domains in Google Oauth (#4226)
Benji Visser [Mon, 26 Nov 2018 09:12:10 +0000 (04:12 -0500)] 
[AIRFLOW-XXX] Remove quotes from domains in Google Oauth (#4226)

Related SO: https://stackoverflow.com/a/52528091/10638329

3 weeks ago[AIRFLOW-3378] KubernetesPodOperator does not delete on timeout failure (#4218)
Victor Noël [Mon, 26 Nov 2018 09:02:08 +0000 (10:02 +0100)] 
[AIRFLOW-3378] KubernetesPodOperator does not delete on timeout failure (#4218)

Signed-off-by: Victor Noel <victor.noel@brennus-analytics.com>
3 weeks ago[AIRFLOW-2642] fix wrong value git-sync initcontainer env GIT_SYNC_ROOT (#3519)
pengc [Mon, 26 Nov 2018 09:00:31 +0000 (17:00 +0800)] 
[AIRFLOW-2642] fix wrong value git-sync initcontainer env GIT_SYNC_ROOT (#3519)

3 weeks ago[AIRFLOW-3365][AIRFLOW-3366] Allow celery_broker_transport_options to be set with...
Victor Villas [Sun, 25 Nov 2018 22:03:04 +0000 (20:03 -0200)] 
[AIRFLOW-3365][AIRFLOW-3366] Allow celery_broker_transport_options to be set with environment variables (#4211)

* [AIRFLOW-3365] Add visibility timeout by key

* [AIRFLOW-3366] Make getsection scan env variables

3 weeks ago[AIRFLOW-3250] Fix for Redis Hook for not authorised connection calls (#4090)
Paweł Graczyk [Sun, 25 Nov 2018 22:01:09 +0000 (23:01 +0100)] 
[AIRFLOW-3250] Fix for Redis Hook for not authorised connection calls (#4090)

Password stay None value and not None (str) in case there is no password set through webadmin interfaces.
This is fix for connections for Redis that not expect autorisation from clients.

3 weeks ago[AIRFLOW-1561] Fix scheduler to pick up example DAGs without other DAGs (#2635)
Shintaro Murakami [Sun, 25 Nov 2018 21:48:11 +0000 (06:48 +0900)] 
[AIRFLOW-1561] Fix scheduler to pick up example DAGs without other DAGs (#2635)

3 weeks ago[AIRFLOW-1196][AIRFLOW-2399] Add templated field in TriggerDagRunOperator (#4228)
Kaxil Naik [Sun, 25 Nov 2018 21:44:07 +0000 (21:44 +0000)] 
[AIRFLOW-1196][AIRFLOW-2399] Add templated field in TriggerDagRunOperator (#4228)

* [AIRFLOW-1196][AIRFLOW-2399] Make trigger_dag_id a templated field for TriggerDagRunOperator

* Update dagrun_operator.py

3 weeks ago[AIRFLOW-2715] Use region setting when launching Dataflow templates (#4139)
Jan Hicken [Sun, 25 Nov 2018 14:10:32 +0000 (15:10 +0100)] 
[AIRFLOW-2715] Use region setting when launching Dataflow templates (#4139)

To launch an instance of a Dataflow template in the configured region,
the API service.projects().locations().teplates() instead of
service.projects().templates() has to be used. Otherwise, all jobs will
always be started in us-central1.

In case there is no region configured, the default region `us-central1`
will get picked up.

To make it even worse, the polling for the job status already honors the
region parameter and will search for the job in the wrong region in the
current implementation. Because the job's status is not found, the
corresponding Airflow task will hang.

3 weeks ago[AIRFLOW-3336] Add new TriggerRule for 0 upstream failures (#4182) 4229/head
rmn36 [Fri, 23 Nov 2018 18:41:04 +0000 (10:41 -0800)] 
[AIRFLOW-3336] Add new TriggerRule for 0 upstream failures (#4182)

Add new TriggerRule that triggers only if all upstream do not fail (success or skipped tasks are allowed)

3 weeks ago[AIRFLOW-1739] Resolve TestDbApiHook naming ambiguity (#2709)
Andy Hadjigeorgiou [Fri, 23 Nov 2018 06:51:41 +0000 (01:51 -0500)] 
[AIRFLOW-1739] Resolve TestDbApiHook naming ambiguity (#2709)

3 weeks ago [AIRFLOW-3348] update run statistics on dag refresh (#4197)
Marcin Szymański [Thu, 22 Nov 2018 22:34:46 +0000 (22:34 +0000)] 
 [AIRFLOW-3348] update run statistics on dag refresh (#4197)

* [AIRFLOW-3348] update run statistics on dag refresh

3 weeks ago[AIRFLOW-3263] Ignore exception when 'run' kills already killed job (#4108)
Jarek Potiuk [Thu, 22 Nov 2018 12:22:35 +0000 (13:22 +0100)] 
[AIRFLOW-3263] Ignore exception when 'run' kills already killed job (#4108)

Sometimes when you run tasks from command line you get exit code = 1 due
to race condition (job runner tries to get process group from the
process that has already been terminated in the meantime)

3 weeks ago[AIRFLOW-3382] Fix incorrect docstring in DatastoreHook (#4222)
Ryan Yuan [Thu, 22 Nov 2018 11:58:32 +0000 (22:58 +1100)] 
[AIRFLOW-3382] Fix incorrect docstring in DatastoreHook (#4222)

Correct docstring in DatastoreHook

3 weeks ago[AIRFLOW-3213] Create ADLS to GCS operator (#4134)
Brandon Kvarda [Wed, 21 Nov 2018 23:18:40 +0000 (15:18 -0800)] 
[AIRFLOW-3213] Create ADLS to GCS operator (#4134)

3 weeks ago[AIRFLOW-3371] BigQueryHook's Ability to Create View (#4213)
Ryan Yuan [Wed, 21 Nov 2018 23:16:18 +0000 (10:16 +1100)] 
[AIRFLOW-3371] BigQueryHook's Ability to Create View (#4213)

3 weeks ago[AIRFLOW-XXX] Update changelog for 1.10.1
Ash Berlin-Taylor [Wed, 21 Nov 2018 22:11:00 +0000 (22:11 +0000)] 
[AIRFLOW-XXX] Update changelog for 1.10.1

3 weeks ago[AIRFLOW-XXX] Update NOTICE file per suggestion (#4220)
Tao Feng [Wed, 21 Nov 2018 22:08:15 +0000 (14:08 -0800)] 
[AIRFLOW-XXX] Update NOTICE file per suggestion (#4220)

3 weeks ago[AIRFLOW-3380] Add metrics documentation (#4219)
Bartosz Ługowski [Wed, 21 Nov 2018 17:44:43 +0000 (18:44 +0100)] 
[AIRFLOW-3380] Add metrics documentation (#4219)

3 weeks ago[AIRFLOW-3375] Support returning multiple tasks with BranchPythonOperator (#4215)
BasPH [Wed, 21 Nov 2018 14:59:19 +0000 (15:59 +0100)] 
[AIRFLOW-3375] Support returning multiple tasks with BranchPythonOperator (#4215)

4 weeks ago[AIRFLOW-XXX] Better instructions for airflow flower (#4214)
Gabriel Nicolas Avellaneda [Tue, 20 Nov 2018 23:13:02 +0000 (21:13 -0200)] 
[AIRFLOW-XXX] Better instructions for airflow flower (#4214)

* Better instructions for airflow flower

It is not clear in the documentation that you need to have flower installed to successful run airflow flower. If you don't have flower installed, running airflow flower will show the following error which is not of much help:

airflow flower
[2018-11-20 17:01:14,836] {__init__.py:51} INFO - Using executor SequentialExecutor
Traceback (most recent call last):
  File "/mnt/secondary/workspace/f4/typo-backend/pipelines/model-pipeline/airflow/bin/airflow", line 32, in <module>
    args.func(args)
  File "/mnt/secondary/workspace/f4/typo-backend/pipelines/model-pipeline/airflow/lib/python3.6/site-packages/airflow/utils/cli.py", line
 74, in wrapper
    return f(*args, **kwargs)
  File "/mnt/secondary/workspace/f4/typo-backend/pipelines/model-pipeline/airflow/lib/python3.6/site-packages/airflow/bin/cli.py", line 1
221, in flower
    broka, address, port, api, flower_conf, url_prefix])
  File "/mnt/secondary/workspace/f4/typo-backend/pipelines/model-pipeline/airflow/lib/python3.6/os.py", line 559, in execvp
    _execvpe(file, args)
  File "/mnt/secondary/workspace/f4/typo-backend/pipelines/model-pipeline/airflow/lib/python3.6/os.py", line 604, in _execvpe
    raise last_exc.with_traceback(tb)
  File "/mnt/secondary/workspace/f4/typo-backend/pipelines/model-pipeline/airflow/lib/python3.6/os.py", line 594, in _execvpe
    exec_func(fullname, *argrest)                                                                                                        FileNotFoundError: [Errno 2] No such file or directory

* Update use-celery.rst

4 weeks ago[AIRFLOW-3308] Fix plugins import (#4153)
John Cheng [Tue, 20 Nov 2018 11:01:16 +0000 (19:01 +0800)] 
[AIRFLOW-3308] Fix plugins import (#4153)

Revert #3906 partially.

4 weeks ago[AIRFLOW-2966] Catch ApiException in the Kubernetes Executor (#4209)
johnhofman [Tue, 20 Nov 2018 10:13:53 +0000 (11:13 +0100)] 
[AIRFLOW-2966] Catch ApiException in the Kubernetes Executor (#4209)

Creating a pod that exceeds a namespace's resource quota throws an
ApiException. This change catches the exception and the task is
re-queued inside the Executor instead of killing the scheduler.

4 weeks ago[AIRFLOW-3359] Add option to pass customer encryption keys to Dataproc (#4200)
Bob Lee [Tue, 20 Nov 2018 09:59:39 +0000 (04:59 -0500)] 
[AIRFLOW-3359] Add option to pass customer encryption keys to Dataproc (#4200)

4 weeks agoAIRFLOW-XXX Fix copy&paste mistake (#4212)
Nicholas Huang [Tue, 20 Nov 2018 09:15:32 +0000 (01:15 -0800)] 
AIRFLOW-XXX Fix copy&paste mistake (#4212)

In emr_create_job_flow_operator.py the :type clearly mismatches with
the :param name, suggesting a copy&paste mistake.

4 weeks ago[AIRFLOW-XXX] Don't publish md5 sigs as part of release (#4210)
Ash Berlin-Taylor [Mon, 19 Nov 2018 18:26:18 +0000 (18:26 +0000)] 
[AIRFLOW-XXX] Don't publish md5 sigs as part of release (#4210)

Apache recommend against publishing MD5 files now as they are relatively
easy to collide and shouldn't be trusted anymore

4 weeks ago[AIRFLOW-3361] Log the task_id in the PendingDeprecationWarning from BaseOperator...
Dr Martin Black [Sun, 18 Nov 2018 21:12:05 +0000 (21:12 +0000)] 
[AIRFLOW-3361] Log the task_id in the PendingDeprecationWarning from BaseOperator (#4030)

Log the task_id in the PendingDeprecationWarning when passing invalid arguments.

4 weeks agoRevert "[AIRFLOW-2966] Catch ApiException in the Kubernetes Executor (#3960)"
Fokko Driesprong [Sun, 18 Nov 2018 17:51:39 +0000 (18:51 +0100)] 
Revert "[AIRFLOW-2966] Catch ApiException in the Kubernetes Executor (#3960)"

This reverts commit 03de9ee155b79f83c3d78c77af66c7d07572dc4b.

4 weeks ago[AIRFLOW-1252] API accept JSON when invoking a trigger dag (#2334)
Sergio Hg [Sun, 18 Nov 2018 17:34:19 +0000 (18:34 +0100)] 
[AIRFLOW-1252] API accept JSON when invoking a trigger dag (#2334)

4 weeks ago[AIRFLOW-2966] Catch ApiException in the Kubernetes Executor (#3960)
johnhofman [Sun, 18 Nov 2018 17:13:54 +0000 (18:13 +0100)] 
[AIRFLOW-2966] Catch ApiException in the Kubernetes Executor (#3960)

Creating a pod that exceeds a namespace's resource quota throws an
ApiException. This change catches the exception and the task is
re-queued inside the Executor instead of killing the scheduler.

4 weeks ago[AIRFLOW-XXX] Update readme for lyft (#4208)
Tao Feng [Sun, 18 Nov 2018 05:46:19 +0000 (21:46 -0800)] 
[AIRFLOW-XXX] Update readme for lyft (#4208)

4 weeks ago[AIRFLOW-XXX] Remove spots in all Airflow logos (#4206)
BasPH [Sat, 17 Nov 2018 18:51:02 +0000 (19:51 +0100)] 
[AIRFLOW-XXX] Remove spots in all Airflow logos (#4206)

4 weeks ago[AIRFLOW-3332] Add method to allow inserting rows into BQ table (#4179)
Ryan Yuan [Sat, 17 Nov 2018 11:52:03 +0000 (22:52 +1100)] 
[AIRFLOW-3332] Add method to allow inserting rows into BQ table (#4179)

4 weeks ago[AIRFLOW-3233] Fix deletion of DAGs in the UI (#4069)
aoen [Sat, 17 Nov 2018 11:35:37 +0000 (06:35 -0500)] 
[AIRFLOW-3233] Fix deletion of DAGs in the UI (#4069)

4 weeks agoRevert "[AIRFLOW-3353] Upgrade redis client (#4203)" (#4205)
Ash Berlin-Taylor [Sat, 17 Nov 2018 11:30:24 +0000 (11:30 +0000)] 
Revert "[AIRFLOW-3353] Upgrade redis client (#4203)" (#4205)

This reverts commit ae629872c5d3d616e189ff9d6d2c8948de66a077.

4 weeks ago[AIRFLOW-3353] Upgrade redis client (#4203)
Joshua Carp [Sat, 17 Nov 2018 07:38:39 +0000 (02:38 -0500)] 
[AIRFLOW-3353] Upgrade redis client (#4203)

4 weeks ago[AIRFLOW-XXX] Add Etsy to companies list (#4204)
Sid Anand [Sat, 17 Nov 2018 07:30:54 +0000 (23:30 -0800)] 
[AIRFLOW-XXX] Add Etsy to companies list (#4204)

4 weeks ago[AIRFLOW-XXX] Fix incorrect URL for Task Tries and Task Duration (#4202)
Kaxil Naik [Sat, 17 Nov 2018 05:12:19 +0000 (05:12 +0000)] 
[AIRFLOW-XXX] Fix incorrect URL for Task Tries and Task Duration (#4202)

4 weeks ago[AIRFLOW-3346] Add hook and operator for GCP transfer service (#4189)
Joshua Carp [Fri, 16 Nov 2018 23:54:50 +0000 (18:54 -0500)] 
[AIRFLOW-3346] Add hook and operator for GCP transfer service (#4189)

4 weeks ago[AIRFLOW-3352] Fix expose_config not honoured on RBAC UI (#4194)
phanindhra [Fri, 16 Nov 2018 23:53:09 +0000 (05:23 +0530)] 
[AIRFLOW-3352] Fix expose_config not honoured on RBAC UI (#4194)

4 weeks ago[AIRFLOW-3355] Fix BigQueryCursor.execute to work with Python3 (#4198)
Kengo Seki [Fri, 16 Nov 2018 22:31:24 +0000 (14:31 -0800)] 
[AIRFLOW-3355] Fix BigQueryCursor.execute to work with Python3 (#4198)

BigQueryCursor.execute uses dict.iteritems internally,
so it fails with Python3 if binding parameters are
provided. This PR fixes this problem.

4 weeks ago[AIRFLOW-3251] KubernetesPodOperator now uses 'image_pull_secrets' argument when...
Victor Noël [Fri, 16 Nov 2018 13:20:42 +0000 (14:20 +0100)] 
[AIRFLOW-3251] KubernetesPodOperator now uses 'image_pull_secrets' argument when creating Pods (#4188)