7 weeks agoFix breeze builds to work after removal of images from CI v1-10-test
Jarek Potiuk [Fri, 13 May 2022 09:58:01 +0000 (11:58 +0200)] 
Fix breeze builds to work after removal of images from CI

12 months agoRelease upgrade_check 1.4.0 16441/head upgrade-check/1.4.0rc1
Kaxil Naik [Mon, 14 Jun 2021 22:13:34 +0000 (23:13 +0100)] 
Release upgrade_check 1.4.0

12 months agofixup! Add conf not importable from airflow rule (#14400)
Kaxil Naik [Mon, 14 Jun 2021 22:08:49 +0000 (23:08 +0100)] 
fixup! Add conf not importable from airflow rule (#14400)

12 months agoAdd `flask_oauth` to to `devel_all`
Kaxil Naik [Sun, 4 Apr 2021 22:09:27 +0000 (23:09 +0100)] 
Add `flask_oauth` to to `devel_all`

This package was missed in `deve_all` hence was missed in constraints file

12 months agoPin SQLAlchemy to <1.4 due to breakage of sqlalchemy-utils (#14812)
Jarek Potiuk [Mon, 15 Mar 2021 20:28:06 +0000 (21:28 +0100)] 
Pin SQLAlchemy to <1.4 due to breakage of sqlalchemy-utils (#14812)

The 1.4 releae of SQLAlchemy breaks sqlalchemy-utils.

This change pins it to < 1.4

Fixes #14811

12 months agoFix too specific parsing of `False` in LegacyUIDeprecated (#14967)
Marian Cepok [Thu, 10 Jun 2021 17:49:45 +0000 (19:49 +0200)] 
Fix too specific parsing of `False` in LegacyUIDeprecated (#14967)

14 months agoSkip `DatabaseVersionCheckRule` check if invalid version is detected (#15122)
Kaxil Naik [Fri, 2 Apr 2021 22:59:43 +0000 (23:59 +0100)] 
Skip `DatabaseVersionCheckRule` check if invalid version is detected (#15122)

When the version is something like `12.2 (Debian 12.2-2.pgdg100+1)`, the upgrade check errors with the following:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/airflow/upgrade/", line 130, in __main__
  File "/usr/local/lib/python3.7/site-packages/airflow/upgrade/", line 118, in run
    all_problems = check_upgrade(formatter, rules)
  File "/usr/local/lib/python3.7/site-packages/airflow/upgrade/", line 38, in check_upgrade
    rule_status = RuleStatus.from_rule(rule)
  File "/usr/local/lib/python3.7/site-packages/airflow/upgrade/", line 44, in from_rule
    result = rule.check()
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/", line 74, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/upgrade/rules/", line 52, in check
    installed_postgres_version = Version(session.execute('SHOW server_version;').scalar())
  File "/usr/local/lib/python3.7/site-packages/packaging/", line 298, in __init__
    raise InvalidVersion("Invalid version: '{0}'".format(version))
packaging.version.InvalidVersion: Invalid version: '12.2 (Debian 12.2-2.pgdg100+1)'

This commit will SKIP the check during such occasions

15 months agoFix pool set and pool delete via cli (#15056)
Aaditya Sharma [Sun, 28 Mar 2021 14:07:40 +0000 (19:37 +0530)] 
Fix pool set and pool delete via cli (#15056)

Co-authored-by: Aaditya Sharma <>
15 months agoFix running "upgrade_check" command in a PTY. (#14977)
Kosteev Eugene [Fri, 26 Mar 2021 12:30:15 +0000 (14:30 +0200)] 
Fix running "upgrade_check" command in a PTY. (#14977)

In case of running in a PTY, the get_terminal_size() call can return (0, 0).
As a consequence "upgrade_check" command fails during formatting of output.

File "/usr/local/lib/airflow/airflow/upgrade/", line 109, in on_next_rule_status
    print(status_line_fmt.format(rule_status.rule.title, status))
ValueError: Sign not allowed in string format specifier

This change is to fallback in case of (0, 0) and prevent command from failing.

15 months agoUpgrade rule to suggest to rename max_threads to parsing_processes (#14913)
Marian Cepok [Tue, 23 Mar 2021 12:29:28 +0000 (13:29 +0100)] 
Upgrade rule to suggest to rename max_threads to parsing_processes (#14913)

This adds an upgrade rule as described in #12672 to rename the max_threads configuration to parsing_processes.

I created this rule on the example of the LoggingConfigurationRule, which doesn't suggest any changes if the old configuration exists but has the default value. I assume, that the idea behind this is, that if the configuration is not set by the user, a default configuration is taken. However, if a user explicitly copied the default configuration in his or her own configuration, in my opinion, a change should be suggested but isn't. But maybe I misunderstood something.

closes #12672

15 months agoAdd conf not importable from airflow rule (#14400)
Sunday Mgbogu [Sun, 14 Mar 2021 10:48:19 +0000 (11:48 +0100)] 
Add conf not importable from airflow rule (#14400)

Closes: #13945

15 months agoAdd Changelog & for 1.10.15 14747/head 1.10.15 1.10.15rc1
Kaxil Naik [Fri, 12 Mar 2021 16:55:42 +0000 (16:55 +0000)] 
Add Changelog & for 1.10.15

15 months agoBump Airflow Version to 1.10.15
Kaxil Naik [Fri, 12 Mar 2021 15:56:45 +0000 (15:56 +0000)] 
Bump Airflow Version to 1.10.15

15 months agoWebserver: Sanitize string passed to origin param (#14738)
Kaxil Naik [Fri, 12 Mar 2021 15:25:02 +0000 (15:25 +0000)] 
Webserver: Sanitize string passed to origin param (#14738)

(cherry-picked from 409c249121bd9c8902fc2ba551b21873ab41f953)

15 months agoDon't exit upgrade_check with code 1 if there are no problems (#14740)
Marian Cepok [Fri, 12 Mar 2021 13:48:39 +0000 (14:48 +0100)] 
Don't exit upgrade_check with code 1 if there are no problems (#14740)

Co-authored-by: Marian Cepok <>
15 months agoMake rbac_app's db.session use the same timezone with @provide_session (#14025) 14705/head
zlhsmfj [Thu, 4 Feb 2021 11:09:51 +0000 (19:09 +0800)] 
Make rbac_app's db.session use the same timezone with @provide_session (#14025)

15 months agoBump ini from 1.3.5 to 1.3.8 in /airflow/www_rbac
Kaxil Naik [Mon, 21 Dec 2020 19:45:30 +0000 (19:45 +0000)] 
Bump ini from 1.3.5 to 1.3.8 in /airflow/www_rbac

15 months agoBump from 1.10.21 to 1.10.23 in /airflow/www_rbac
snyk-bot [Sat, 19 Dec 2020 04:19:22 +0000 (04:19 +0000)] 
Bump from 1.10.21 to 1.10.23 in /airflow/www_rbac

The following vulnerabilities are fixed with an upgrade:

15 months agoChange the format for sha512 sum for releases (#12867)
Kaxil Naik [Sun, 6 Dec 2020 23:46:06 +0000 (23:46 +0000)] 
Change the format for sha512 sum for releases (#12867)


This format is popular and supported by tools like Ansible.

❯ sha512sum ../airflow-dev/1.10.14rc3/apache-airflow-1.10.14rc3-bin.tar.gz
953d3c04ee6fd2fa96e126750e642fc0872add96d180901440a91bd61c494a711b48836c634d93dcb181006935772556d5b4426671bf1a638f0a0698b51b119f  ../airflow-dev/1.10.14rc3/apache-airflow-1.10.14rc3-bin.tar.gz


❯ gpg --print-md SHA512 ../airflow-dev/1.10.14rc3/apache-airflow-1.10.14rc3-bin.tar.gz
953D3C04 EE6FD2FA 96E12675 0E642FC0 872ADD96 D1809014 40A91BD6 1C494A71 1B48836C
 634D93DC B1810069 35772556 D5B44266 71BF1A63 8F0A0698 B51B119F

(cherry picked from commit a00f25011fc6c859b27b6c78b9201880cf6323ce)

15 months agoSign release files with an key by default (#12241)
Ash Berlin-Taylor [Tue, 10 Nov 2020 11:26:48 +0000 (11:26 +0000)] 
Sign release files with an key by default (#12241)

If you have more than a single private key in your GPG trust store, gpg
will use the first one, which for me is not right.

This changes the script to by default use any key with `` in
the name. This is a patch I've been carrying locally for about 8
releases now :D

(cherry picked from commit 3ab5828142b8ae786556867552d92e7e61ae2fb1)

15 months agoRelease upgrade_check 1.3.0 upgrade-check/1.3.0 upgrade-check/1.3.0rc2
Kaxil Naik [Tue, 9 Mar 2021 22:57:59 +0000 (22:57 +0000)] 
Release upgrade_check 1.3.0

15 months agoFix wrong warning about class that was not used in a dag file (#14700)
Ephraim Anierobi [Wed, 10 Mar 2021 19:51:00 +0000 (20:51 +0100)] 
Fix wrong warning about class that was not used in a dag file (#14700)

Running upgrade_check with this import in a dag:
from airflow.operators.slack_operator import SlackAPIPostOperator

Causes these warnings:
[1] Using `airflow.operators.slack_operator.SlackAPIPostOperator` should be replaced by `airflow.providers.slack.operators.slack.SlackAPIPostOperator`.
[2] Using `airflow.operators.slack_operator.SlackAPIOperator` should be replaced by `airflow.providers.slack.operators.slack.SlackAPIOperator`

Only 1 above is correct.
This PR fixes it by checking that both the old import part and class name exists in the dag before issuing warning

15 months agoFix webserver exiting when gunicorn master crashes (#13470)
drago-f5a [Tue, 9 Mar 2021 22:30:45 +0000 (16:30 -0600)] 
Fix webserver exiting when gunicorn master crashes (#13470)

closes #13469

15 months agoFill DagBag from dag_folder setting for upgrade rules (#14588)
kenfinster [Tue, 9 Mar 2021 22:28:20 +0000 (00:28 +0200)] 
Fill DagBag from dag_folder setting for upgrade rules (#14588)

For some reason: BaseOperatorMetaclassRule does not consider the dag folder set by the user. This causes issues when the user needs to set the dag folder with environment variable.

15 months agoFix merging of secrets and configmaps (#14090)
cris-b [Tue, 9 Mar 2021 20:06:42 +0000 (20:06 +0000)] 
Fix merging of secrets and configmaps (#14090)

closes #13023

15 months agoBugfix: False positives for Custom Executors via Plugins check (#14680)
Kaxil Naik [Tue, 9 Mar 2021 15:21:37 +0000 (15:21 +0000)] 
Bugfix: False positives for Custom Executors via Plugins check (#14680)


15 months agoFix Failing test on v1-10-stable (#14678)
Kaxil Naik [Tue, 9 Mar 2021 15:21:21 +0000 (15:21 +0000)] 
Fix Failing test on v1-10-stable (#14678)

Some of the tests were irrelevant in Python 2.7 because of unicode differences

15 months agoBugfix: Fix False alarm in import changes rule (#14493)
Ephraim Anierobi [Wed, 3 Mar 2021 00:34:31 +0000 (01:34 +0100)] 
Bugfix: Fix False alarm in import changes rule (#14493)

Closes: #14295

16 months agoUse CustomSQLAInterface instead of SQLAInterface (#14475)
Dr. Dennis Akpenyi [Sat, 27 Feb 2021 01:29:50 +0000 (02:29 +0100)] 
Use CustomSQLAInterface instead of SQLAInterface (#14475)

* Use CustomSQLAInterface instead of SQLAInterface
for custom data models.

From Airflow 2.0, if you want to define your own Flask App Builder
models you need to use CustomSQLAInterface instead of SQLAInterface.

For Non-RBAC replace:

`from flask_appbuilder.models.sqla.interface import SQLAInterface`
`datamodel = SQLAInterface(your_data_model)`

with RBAC (in 1.10):

`from airflow.www_rbac.utils import CustomSQLAInterface`
`datamodel = CustomSQLAInterface(your_data_model)`

and in 2.0:

`from airflow.www.utils import CustomSQLAInterface`
`datamodel = CustomSQLAInterface(your_data_model)`

closes #13226

16 months agoBugFix: fix the delete_dag function of json_client (#14441)
花儿黑马 [Fri, 26 Feb 2021 12:12:16 +0000 (20:12 +0800)] 
BugFix: fix the delete_dag function of json_client (#14441)

closes #14373

16 months agoFixed deprecation message for "variables" command (#14457)
Bruno Guimarães [Fri, 26 Feb 2021 05:10:57 +0000 (02:10 -0300)] 
Fixed deprecation message for "variables" command (#14457)

16 months agoAdd 'airflow variables list' command for 1.10.x transition version (#14462)
Xiaodong DENG [Fri, 26 Feb 2021 05:09:58 +0000 (06:09 +0100)] 
Add 'airflow variables list' command for 1.10.x transition version (#14462)

Helps ease transition from 1.10.x to 2.0

16 months agoFix comparing airflow version to work with older versions of packaging library (...
eladkal [Thu, 25 Feb 2021 01:28:11 +0000 (03:28 +0200)] 
Fix comparing airflow version to work with older versions of packaging library (#14435)

closes: #14359

16 months agoFix Incorrect warning in upgrade check and error in reading file (#14344)
Ephraim Anierobi [Wed, 24 Feb 2021 19:14:23 +0000 (20:14 +0100)] 
Fix Incorrect warning in upgrade check and error in reading file (#14344)

Closes: #14340, #14258, #14243

16 months agoFix permission error on non-POSIX filesystem (#13121) (#14383)
cris-b [Tue, 23 Feb 2021 21:46:34 +0000 (21:46 +0000)] 
Fix permission error on non-POSIX filesystem (#13121) (#14383)

* Fix
 - failure to change ownership of log file on azure

16 months agoHandle possible suffix in MySQL version + avoid hard-coding (#14274)
Xiaodong DENG [Thu, 18 Feb 2021 18:44:59 +0000 (19:44 +0100)] 
Handle possible suffix in MySQL version + avoid hard-coding (#14274)

closes #14262, and do additional minor improvement

16 months agoFix `airflow tasks clear` cli command wirh `--yes` (#14188)
Ephraim Anierobi [Wed, 17 Feb 2021 00:34:05 +0000 (01:34 +0100)] 
Fix `airflow tasks clear` cli command wirh `--yes` (#14188)

Closes: #14171

16 months agoRelease ``upgrade_check`` 1.2.0 (#14140) 14333/head upgrade-check/1.2.0 upgrade-check/1.2.0rc1
Kaxil Naik [Mon, 8 Feb 2021 23:35:33 +0000 (23:35 +0000)] 
Release ``upgrade_check`` 1.2.0 (#14140)

- Added Changelog
- Updated Version

16 months agoAdd Version command for Upgrade Check (#12929)
Kaxil Naik [Mon, 8 Feb 2021 22:16:27 +0000 (22:16 +0000)] 
Add Version command for Upgrade Check (#12929)

❯ airflow upgrade_check --version

Help command:

❯ airflow upgrade_check --help
usage: airflow upgrade_check [-h] [-s SAVE] [-i IGNORE] [-c CONFIG] [-l] [-V]

optional arguments:
  -h, --help            show this help message and exit
  -s SAVE, --save SAVE  Saves the result to the indicated file. The file
                        format is determined by the file extension.
  -i IGNORE, --ignore IGNORE
                        Ignore a rule. Can be used multiple times.
  -c CONFIG, --config CONFIG
                        Path to upgrade check config yaml file.
  -l, --list            List the upgrade checks and their class names
  -V, --version         Show the version of upgrade_check

16 months agoUse Tabular Format for the List of Upgrade Check Rules (#14139)
Kaxil Naik [Mon, 8 Feb 2021 22:00:13 +0000 (22:00 +0000)] 
Use Tabular Format for the List of Upgrade Check Rules (#14139)



❯ airflow upgrade_check --list

Upgrade Checks:
- VersionCheckRule: Check for latest versions of apache-airflow and checker
- AirflowMacroPluginRemovedRule: Remove airflow.AirflowMacroPlugin class
- BaseOperatorMetaclassRule: Ensure users are not using custom metaclasses in custom operators
- ChainBetweenDAGAndOperatorNotAllowedRule: Chain between DAG and operator not allowed.
- ConnTypeIsNotNullableRule: Connection.conn_type is not nullable
- CustomExecutorsRequireFullPathRule: Custom Executors now require full path
- DatabaseVersionCheckRule: Check versions of PostgreSQL, MySQL, and SQLite to ease upgrade to Airflow 2.0
- DbApiRule: Hooks that run DB functions must inherit from DBApiHook
- FernetEnabledRule: Fernet is enabled by default
- GCPServiceAccountKeyRule: GCP service account key deprecation
- HostnameCallable: Unify hostname_callable option in core section
- ImportChangesRule: Changes in import paths of hooks, operators, sensors and others
- LegacyUIDeprecated: Legacy UI is deprecated by default
- LoggingConfigurationRule: Logging configuration has been moved to new section
- MesosExecutorRemovedRule: Removal of Mesos Executor
- NoAdditionalArgsInOperatorsRule: No additional argument allowed in BaseOperator.
- PodTemplateFileRule: Users must set a kubernetes.pod_template_file value
- SendGridEmailerMovedRule: SendGrid email uses old airflow.contrib module
- SparkJDBCOperatorConnIdRule: Check Spark JDBC Operator default connection name
- TaskHandlersMovedRule: Changes in import path of remote task handlers
- UniqueConnIdRule: Connection.conn_id is not unique

❯ airflow upgrade_check --list

Rule Name                                 Description
----------------------------------------  ------------------------------------------------------------------------------
VersionCheckRule                          Check for latest versions of apache-airflow and checker
AirflowMacroPluginRemovedRule             Remove airflow.AirflowMacroPlugin class
BaseOperatorMetaclassRule                 Ensure users are not using custom metaclasses in custom operators
ChainBetweenDAGAndOperatorNotAllowedRule  Chain between DAG and operator not allowed.
ConnTypeIsNotNullableRule                 Connection.conn_type is not nullable
CustomExecutorsRequireFullPathRule        Custom Executors now require full path
DatabaseVersionCheckRule                  Check versions of PostgreSQL, MySQL, and SQLite to ease upgrade to Airflow 2.0
DbApiRule                                 Hooks that run DB functions must inherit from DBApiHook
FernetEnabledRule                         Fernet is enabled by default
GCPServiceAccountKeyRule                  GCP service account key deprecation
HostnameCallable                          Unify hostname_callable option in core section
ImportChangesRule                         Changes in import paths of hooks, operators, sensors and others
LegacyUIDeprecated                        Legacy UI is deprecated by default
LoggingConfigurationRule                  Logging configuration has been moved to new section
MesosExecutorRemovedRule                  Removal of Mesos Executor
NoAdditionalArgsInOperatorsRule           No additional argument allowed in BaseOperator.
PodTemplateFileRule                       Users must set a kubernetes.pod_template_file value
SendGridEmailerMovedRule                  SendGrid email uses old airflow.contrib module
SparkJDBCOperatorConnIdRule               Check Spark JDBC Operator default connection name
TaskHandlersMovedRule                     Changes in import path of remote task handlers
UniqueConnIdRule                          Connection.conn_id is not unique


16 months agoFix broken ``airflow upgrade_check`` command (#14137)
Kaxil Naik [Mon, 8 Feb 2021 20:35:35 +0000 (20:35 +0000)] 
Fix broken ``airflow upgrade_check`` command (#14137) broke the ability
to run ``airflow upgrade_check`` command.

When ``airflow upgrade_check`` is run, the airflow cli runs
`airflow.upgrade.checker.register_arguments` function.

This function registers all arguments and sets the default function to run.

Since we don't want to handle the logic in ``airflow``, we have a hack
to only run anything via that ``run`` command. So the logic about
checking for args should be in it. removed ``subparser.set_defaults(func=run)``.

Due to which I get the following:


❯ airflow upgrade_check --list

Please install apache-airflow-upgrade-check distribution from PyPI to perform upgrade checks

Because default is not set in ``, it uses one set in

and hence just prints `Please install apache-airflow-upgrade-check distribution from PyPI to perform upgrade checks`.

**After the fix in this PR**:

❯ airflow upgrade_check --list

Upgrade Checks:
- VersionCheckRule: Check for latest versions of apache-airflow and checker
- AirflowMacroPluginRemovedRule: Remove airflow.AirflowMacroPlugin class
- BaseOperatorMetaclassRule: Ensure users are not using custom metaclasses in custom operators
- ChainBetweenDAGAndOperatorNotAllowedRule: Chain between DAG and operator not allowed.
- ConnTypeIsNotNullableRule: Connection.conn_type is not nullable
- CustomExecutorsRequireFullPathRule: Custom Executors now require full path
- DatabaseVersionCheckRule: Check versions of PostgreSQL, MySQL, and SQLite to ease upgrade to Airflow 2.0
- DbApiRule: Hooks that run DB functions must inherit from DBApiHook
- FernetEnabledRule: Fernet is enabled by default
- GCPServiceAccountKeyRule: GCP service account key deprecation
- HostnameCallable: Unify hostname_callable option in core section
- ImportChangesRule: Changes in import paths of hooks, operators, sensors and others
- LegacyUIDeprecated: Legacy UI is deprecated by default
- LoggingConfigurationRule: Logging configuration has been moved to new section
- MesosExecutorRemovedRule: Removal of Mesos Executor
- NoAdditionalArgsInOperatorsRule: No additional argument allowed in BaseOperator.
- PodTemplateFileRule: Users must set a kubernetes.pod_template_file value
- SendGridEmailerMovedRule: SendGrid email uses old airflow.contrib module
- SparkJDBCOperatorConnIdRule: Check Spark JDBC Operator default connection name
- TaskHandlersMovedRule: Changes in import path of remote task handlers
- UniqueConnIdRule: Connection.conn_id is not unique

16 months ago[kubernetes_generate_dag_yaml] - Fix dag yaml generate func (#13816) 14069/head
Lord-Y [Thu, 4 Feb 2021 19:11:50 +0000 (20:11 +0100)] 
[kubernetes_generate_dag_yaml] - Fix dag yaml generate func (#13816)

Signed-off-by: Lord-Y <>
16 months agoCreated DatabaseVersionCheckRule class (#13955)
Dr. Dennis Akpenyi [Wed, 3 Feb 2021 21:34:26 +0000 (22:34 +0100)] 
Created DatabaseVersionCheckRule class (#13955)

This upgrade check inspects the version of the supported database
backend (PostgreSQL, MySQL, and SQLite) so as to verify if the version
is supported in Airflow 2.0

This is ease upgrade to Airflow 2.0

closes: #13850

16 months agoTreat default value in HostnameCallable rule as good one (#13670)
Tomek Urbaszek [Tue, 2 Feb 2021 13:29:07 +0000 (14:29 +0100)] 
Treat default value in HostnameCallable rule as good one (#13670)

16 months agoCreated rule for SparkJDBCOperator class conn_id (#13798)
Dr. Dennis Akpenyi [Tue, 2 Feb 2021 11:26:43 +0000 (12:26 +0100)] 
Created rule for SparkJDBCOperator class conn_id (#13798)

This PR implements an upgrade check for SparkJDBCOperator.

In Airflow 1.10.x, the default name for SparkJDBCOperator class conn_id
is 'spark-default'. From Airflow 2.0, it has been changed to
'spark_default' to conform with the naming conventions of all
other connection names.

closes: #12918

17 months agoUpdate Mongodb inventory URL to fix docs build (#13939) (#13991)
Kaxil Naik [Sat, 30 Jan 2021 20:19:30 +0000 (20:19 +0000)] 
Update Mongodb inventory URL to fix docs build (#13939) (#13991)

Master is failing on docs build because of the following error:

Failed to fetch inventory:

This is because the URL is changed to

(cherry picked from commit 7a5aafce08374c75562e3eb728413fefc4ab6e01)

17 months agoUpgrade tool: skip bad file test for Python 2.7 (#13981)
Madison Bowden [Sat, 30 Jan 2021 00:11:46 +0000 (16:11 -0800)] 
Upgrade tool: skip bad file test for Python 2.7 (#13981)

17 months agoRemove UndefinedJinjaVariableRule 13897/head
Jarek Potiuk [Sat, 23 Jan 2021 04:10:29 +0000 (05:10 +0100)] 
Remove UndefinedJinjaVariableRule

This rule is problematic because it hangs randomly - both in test
and production.

This PR removes the rule.

17 months agoRestores accidentally removed kerberos authenttication
Jarek Potiuk [Sat, 23 Jan 2021 03:09:22 +0000 (04:09 +0100)] 
Restores accidentally removed kerberos authenttication

17 months agoAdd clearer exception for read failures in macro plugin upgrade check (#13371)
Madison Bowden [Sat, 23 Jan 2021 00:04:07 +0000 (16:04 -0800)] 
Add clearer exception for read failures in macro plugin upgrade check (#13371)

Resolves #13349

17 months agoEnsure all statsd timers use millisecond values. (#10633)
Martijn Pieters [Fri, 22 Jan 2021 22:14:40 +0000 (22:14 +0000)] 
Ensure all statsd timers use millisecond values. (#10633)

* Backport pull request #6682 to v1-10
* Backport pull request #10629 to v1-10

17 months agoStreamLogWriter: Provide (no-op) close method (#10885)
Martijn Pieters [Fri, 22 Jan 2021 22:09:20 +0000 (22:09 +0000)] 
StreamLogWriter: Provide (no-op) close method (#10885)

Some contexts try to close their reference to the stderr stream at logging shutdown, this ensures these don't break.

related: #10882, #10884

17 months agoFix test broken by #13392 13822/head
Kaxil Naik [Fri, 22 Jan 2021 21:29:01 +0000 (21:29 +0000)] 
Fix test broken by #13392

17 months agoFix test for Airflow docs (#13561)
Kaxil Naik [Fri, 22 Jan 2021 21:13:22 +0000 (21:13 +0000)] 
Fix test for Airflow docs (#13561)

17 months agoRun "third party" github actions from submodules instead (#13514)
Ash Berlin-Taylor [Mon, 11 Jan 2021 10:38:15 +0000 (10:38 +0000)] 
Run "third party" github actions from submodules instead (#13514)

Rather than having to mirror all the repos we can instead use
git submodules to pull in the third party actions we want to use - with
recent(ish) changes in review for submodules on GitHub we still get the
same "review/audit" visibility for changes, but this way we don't have
to either "pollute" our repo with the actions code, nor do we have to
maintain a fork of the third party action.

(cherry picked from commit f115983550833b1516ce58e66b49fe4fb1a0eff7)

17 months agoDisable persisting credentials in Github Action's checkout (#13389)
Jarek Potiuk [Wed, 30 Dec 2020 13:25:29 +0000 (14:25 +0100)] 
Disable persisting credentials in Github Action's checkout (#13389)

This PR disables persisting credentials in Github Actions checkout.

This is a result of discussion in

It turns out that contrary to the documentation actios (specifically
checkout action) can use GITHUB_TOKEN without specifying it as
input in the yaml file and the GitHub checkout action
leaves the repository with credentials stored locally that
enable pushing to Github Repository by any step in the same
job. This was thought to be forbidden initially (and the
documentation clearly says that the action must have the
GITHUB_TOKEN passed to it in .yaml workflow in order to
use it). But apparently it behaves differently.

This leaves open an attack vector where for example
any PIP package installed in the following steps could push
any changes to GitHub Repository of Apache Airflow.

Security incidents have been reported to both GitHub and
Apache Security team, but in the meantime we add configuration
to remove credentials after checkout step.

> Using the GITHUB_TOKEN in a workflow

> To use the GITHUB_TOKEN secret, you *must* reference it in your workflow
  file. Using a token might include passing the token as an input to an
  action that requires it, or making authenticated GitHub API calls.

(cherry picked from commit d079b913d283378dca37dc9ea25b04186d3e326c)

17 months agoAlso add codecov action to apache airflow repo (#13328)
Jarek Potiuk [Sun, 27 Dec 2020 16:42:45 +0000 (17:42 +0100)] 
Also add codecov action to apache airflow repo (#13328)

Follow up after #13327

(cherry picked from commit 98896e4e327f256fd04087a49a13e16a246022c9)

17 months agoSwitch to Apache-owned GitHub actions (#13327)
Jarek Potiuk [Sun, 27 Dec 2020 16:13:50 +0000 (17:13 +0100)] 
Switch to Apache-owned GitHub actions (#13327)

There was a change in Policy of ASF that only "Made by GitHub"
actions and actions residing in Apache-owned repositories
are allowed to be used for ASF projects. This was in
response to a security incident.

More details:







(cherry picked from commit c6d66cd15fe85d3bca357c9e60ac8434b843e5d6)

17 months agoAllow webserver to read pod logs directly (#12598)
Daniel Imberman [Tue, 24 Nov 2020 23:13:23 +0000 (15:13 -0800)] 
Allow webserver to read pod logs directly (#12598)

* Allow webserver to read pod logs directly

For users who are testing the KubernetesExecutor, allows users to read
pod logs via the Kubernetes API. Worth noting that these logs will only
be accessible while the worker is running.

* fix tests

(cherry picked from commit 9f28e416dbc6374dc9c7115304731a7bc0b4bfa9)

17 months agoUpdate chart readme to remove astronomer references (#13210)
dstandish [Mon, 21 Dec 2020 13:12:19 +0000 (05:12 -0800)] 
Update chart readme to remove astronomer references (#13210)

(cherry picked from commit a3cc78dc16bf228901ba64e65d202d4685bf4fc7)

17 months agoFix parenthesis preventing Keda ScaledObject creation (#13183)
dstandish [Mon, 21 Dec 2020 10:19:26 +0000 (02:19 -0800)] 
Fix parenthesis preventing Keda ScaledObject creation (#13183)

(cherry picked from commit a9d562e1c3c16c98750c9e3be74347f882acb97a)

17 months agoUse new logging options on values.yaml (#13173)
Flávio de Assis [Sun, 20 Dec 2020 01:14:35 +0000 (22:14 -0300)] 
Use new logging options on values.yaml (#13173)

(cherry picked from commit 23a47879ababe76f6cf9034a2bae055b2a91bf1f)

17 months agoThe default value in chart should be 2.0.0 (#13125)
Jarek Potiuk [Fri, 18 Dec 2020 10:50:23 +0000 (11:50 +0100)] 
The default value in chart should be 2.0.0 (#13125)

(cherry picked from commit f5c4b2442d096ccb873d18f50bc2c5d89d780b03)

17 months agoDefault python version is used when building image
Jarek Potiuk [Wed, 23 Dec 2020 14:46:35 +0000 (15:46 +0100)] 
Default python version is used when building image

For image build the python version is passed via
PYTHON_MAJOR_MINOR_VERSION but there is a part of the build
(preparing airflow package) that uses python installed on host.

This is fine for Master/2.0 to use same version as the image
but it should be unified (and in 1.10 when trying to build 2.7
image it would fail).

(cherry picked from commit ba1111a6cc3906480394b22536d5adf8a3846c82)

17 months agoClick should be limited for Python 2.7
Jarek Potiuk [Wed, 23 Dec 2020 06:54:09 +0000 (07:54 +0100)] 
Click should be limited for Python 2.7

17 months agoReset PIP version after eager upgrade (#13251)
Jarek Potiuk [Tue, 22 Dec 2020 13:23:54 +0000 (14:23 +0100)] 
Reset PIP version after eager upgrade (#13251)

PIP upgrades itself after eager update, and since we (for now)
stick with the 20.2.4 version we want to reset PIP to that
version after eager upgrade.

(cherry picked from commit c44092f8df9aa8555ef594d9fae4f28011a3a5a6)

17 months agoInclude airflow/contrib/executors in the dist package
Kaxil Naik [Tue, 22 Dec 2020 15:18:58 +0000 (15:18 +0000)] 
Include airflow/contrib/executors in the dist package

The was removed by mistake in

17 months agoSkip identity pre-commit in ci and always display it first
Jarek Potiuk [Wed, 16 Dec 2020 12:17:50 +0000 (13:17 +0100)] 
Skip identity pre-commit in ci and always display it first

(cherry picked from commit 76f2215b95ab8da0d3c4e821138b6dd4275b7c79)

17 months agoProduction images on CI are now built from packages (#12685)
Jarek Potiuk [Sun, 6 Dec 2020 22:36:33 +0000 (23:36 +0100)] 
Production images on CI are now built from packages (#12685)

So far, the production images of Airflow were using sources
when they were built on CI. This PR changes that, to build
airflow + providers packages first and install them
rather than use sources as installation mechanism.

Part of #12261

(cherry picked from commit ed1825c0264d1f77c4754b722fb3721cbcd779d7)

17 months agoAdd identity pre-commit hook (#13089)
Ruben Laguna [Wed, 16 Dec 2020 08:03:48 +0000 (09:03 +0100)] 
Add identity pre-commit hook (#13089)

(cherry picked from commit 99c2e03da9b552b74d78b1b10ce9b8f85c686678)

17 months agoAllows to install Airflow in Breeze from PIP with configurable extras (#13055)
Jarek Potiuk [Mon, 14 Dec 2020 14:10:06 +0000 (15:10 +0100)] 
Allows to install Airflow in Breeze from PIP with configurable extras (#13055)

The extras configured by --extras Breeze switch are now
passed to pip install command in case airfow is installed via
--install-airflow-version or --install-airflow-reference switch.

(cherry picked from commit 26c685425996a2e8127d349a7b98ec92c74ee3f2)

17 months agoFixes image building in DockerHub (#13039)
Jarek Potiuk [Sun, 13 Dec 2020 11:08:55 +0000 (12:08 +0100)] 
Fixes image building in DockerHub (#13039)

(cherry picked from commit 0d49a4742c269fa537d59432cc4635286430a5a4)

17 months agoInstall airflow and providers from dist and verifies them (#13033)
Jarek Potiuk [Sat, 12 Dec 2020 18:38:30 +0000 (19:38 +0100)] 
Install airflow and providers from dist and verifies them  (#13033)

* Install airflow and providers from dist and verifies them

This check is there to prevent problems similar to those reported
in #13027 and fixed in #13031.

Previously we always built airflow from wheels, only providers were
installed from sdist packages and tested. In this version both
airflow and providers are installed using the same package format
(sdist or wheel).

* Update scripts/in_container/

Co-authored-by: Kaxil Naik <>
Co-authored-by: Kaxil Naik <>
(cherry picked from commit abf2a4264b18e750dbc2eb384a86d08d821dfba4)

17 months agoFix gpg verification command (#13035)
Jarek Potiuk [Sat, 12 Dec 2020 18:30:09 +0000 (19:30 +0100)] 
Fix gpg verification command (#13035)

(cherry picked from commit 825e9cb98411fe3999e79ee66d00e401fc4a91e0)

17 months agoChanges release image preparation to use PyPI packages (#12990)
Jarek Potiuk [Sat, 12 Dec 2020 11:01:58 +0000 (12:01 +0100)] 
Changes release image preparation to use PyPI packages (#12990)

* Changes release image preparation to use PyPI packages

Since we released all teh provider packages to PyPI now in
RC version, we can now change the mechanism to prepare the
production to use released packages in case of tagged builds.

The "branch" production images are still prepared using the
CI images and .whl packages built from sources, but the
release packages are built from officially released PyPI

Also some corrections and updates were made to the release process:

* the constraint tags when RC candidate is sent should contain
  rcn suffix.

* there was missing step about pushing the release tag once the
  release is out

* pushing tag to GitHub should be done after the PyPI packages
  are uploaded, so that automated image building in DockerHub
  can use those packages.

* added a note that in case we will release some provider
  packages that depend on the just released airflow version
  they shoudl be released after airflow is in PyPI but before
  the tag is pushed to GitHub (also to allow the image to be
  build automatically from the released packages)

Fixes: #12970

* Update dev/

Co-authored-by: Ash Berlin-Taylor <>
* Update dev/

Co-authored-by: Ash Berlin-Taylor <>
Co-authored-by: Ash Berlin-Taylor <>
(cherry picked from commit db027735a7ee03d2678c8ca8712dfd75de148261)

17 months agoAdd changes from 1.10.14 (#12993)
Kaxil Naik [Thu, 10 Dec 2020 20:52:29 +0000 (20:52 +0000)] 
Add changes from 1.10.14 (#12993)

This commit adds Changelog, and replaces 1.10.13 to 1.10.14
across the codebase

(cherry picked from commit 969d3ea4f86a39807d696ca6f47c7cdf23526333)

17 months agoUpdate Dockerfile (#12987)
jao6693 [Thu, 10 Dec 2020 20:01:49 +0000 (21:01 +0100)] 
Update Dockerfile (#12987)

Fix permission issue in Azure DevOps when running the script, which prevents the build to succeed

/bin/bash: ./scripts/docker/ Permission denied
The command '/bin/bash -o pipefail -e -u -x -c ./scripts/docker/ dev' returned a non-zero code: 126
##[error]The command '/bin/bash -o pipefail -e -u -x -c ./scripts/docker/ dev' returned a non-zero code: 126
##[error]The process '/usr/bin/docker' failed with exit code 126

(cherry picked from commit 2ec03cd9267f1cfd7fa3fa437ffad7deee3241e8)

17 months agoUpdate (#12988)
jao6693 [Thu, 10 Dec 2020 20:00:41 +0000 (21:00 +0100)] 
Update (#12988)

Fix permission issue in Azure DevOps when running the script, which prevents the build to succeed

/bin/bash: ./scripts/docker/ Permission denied
The command '/bin/bash -o pipefail -e -u -x -c ./scripts/docker/ dev' returned a non-zero code: 126
##[error]The command '/bin/bash -o pipefail -e -u -x -c ./scripts/docker/ dev' returned a non-zero code: 126
##[error]The process '/usr/bin/docker' failed with exit code 126

(cherry picked from commit d84faa36a02f6a6ae2800e601bda4a20c3188190)

17 months agoUpdate CI to run tests againt v2-0-test branch (#10891)
Kaxil Naik [Wed, 9 Dec 2020 19:35:04 +0000 (19:35 +0000)] 
Update CI to run tests againt v2-0-test branch (#10891)

(cherry picked from commit db166ba75c447a08b94e7be1ab09042fd6361581)

17 months agoUpdate CI to run tests againt v2-0-test branch (#10891)
Kaxil Naik [Wed, 9 Dec 2020 19:35:04 +0000 (19:35 +0000)] 
Update CI to run tests againt v2-0-test branch (#10891)

(cherry picked from commit db166ba75c447a08b94e7be1ab09042fd6361581)

17 months agoApply labels to Docker images in a single instruction (#12931)
Ash Berlin-Taylor [Wed, 9 Dec 2020 05:19:38 +0000 (05:19 +0000)] 
Apply labels to Docker images in a single instruction (#12931)

* Apply labels to Docker images in a single instruction

While looking at the build logs for something else I noticed this
oddity at the end of the CI logs:

Tue, 08 Dec 2020 21:20:19 GMT Step 125/135 : LABEL org.apache.airflow.distro="debian"
Tue, 08 Dec 2020 21:21:14 GMT Step 133/135 : LABEL org.apache.airflow.commitSha=${COMMIT_SHA}
Tue, 08 Dec 2020 21:21:14 GMT  ---> Running in 1241a5f6cdb7
Tue, 08 Dec 2020 21:21:21 GMT Removing intermediate container 1241a5f6cdb7

Applying all the labels took 1m2s! Hopefully applying these in a single
layer/command should speed things up.

A less extreme example still took 43s

Tue, 08 Dec 2020 20:44:40 GMT Step 125/135 : LABEL org.apache.airflow.distro="debian"
Tue, 08 Dec 2020 20:45:18 GMT Step 133/135 : LABEL org.apache.airflow.commitSha=${COMMIT_SHA}
Tue, 08 Dec 2020 20:45:18 GMT  ---> Running in dc601207dbcb
Tue, 08 Dec 2020 20:45:23 GMT Removing intermediate container dc601207dbcb
Tue, 08 Dec 2020 20:45:23 GMT  ---> 5aae5dd0f702

* Update Dockerfile

(cherry picked from commit 63ea88d1b1bdad5ce24f498fdf3600217069b4ad)

17 months agoBuilds prod images on DockerHub from packages (#12908)
Jarek Potiuk [Tue, 8 Dec 2020 11:45:03 +0000 (12:45 +0100)] 
Builds prod images on DockerHub from packages (#12908)

This build combines building both CI and PROD image in one
script execution on DockerHub per python version.

First the CI image is build and secondly, the image is used
to build all the packages from sources and then those
packages are used to build the PROD image.

Resulting image will be package image built from latest sources.

Closes: #12261
(cherry picked from commit f9e9ad2b096ff9d8ee78224333f799ca3968b6bd)

17 months agoClarifies version args for installing 1.10 in Docker (#12875)
Jarek Potiuk [Mon, 7 Dec 2020 15:08:55 +0000 (16:08 +0100)] 
Clarifies version args for installing 1.10 in Docker (#12875)

This change clarifies that AIRFLOW_VERSION should be passed
together with AIRFLOW_INSTALL_VERSION when the Docker image
is build.

Fixes #8612

(cherry picked from commit 1dcef78b12ac4746345f4630966e0dc4e32e41da)

17 months agoAdds airflow as viable docker command in official image (#12878)
Jarek Potiuk [Mon, 7 Dec 2020 14:17:49 +0000 (15:17 +0100)] 
Adds airflow as viable docker command in official image (#12878)

The change is backwards-compatible. It still allows to pass airflow
command without "airflow" as first parameter, but you can now
also pass "airflow" and the rest of the parameters will
be treated as "airflow" command parameters.

Documentation is updated to reflect the entrypoint behaviour
including _CMD option in SQL connections.

Part of #12762 and #12602

Partially extracted from  #12766

(cherry picked from commit 4d44faac77b639a19379da714bf532ceb9416a1b)

17 months agoArtifacts in Github Action have a short retention period (#12793)
Kamil Breguła [Fri, 4 Dec 2020 07:37:38 +0000 (08:37 +0100)] 
Artifacts in Github Action have a short retention period (#12793)

(cherry picked from commit fa4fc37b966b331377257fe964824914524dba96)

17 months agoFix chart jobs delete policy for improved idempotency (#12646)
Florent Chehab [Tue, 1 Dec 2020 18:24:42 +0000 (19:24 +0100)] 
Fix chart jobs delete policy for improved idempotency (#12646)

The chart has two jobs (migrate-database & create-user).
These jobs are run post-install and post-upgrade and only deleted on success.

So if for some reason (quick reinstall / upgrade), the job fails or is stuck then helm
will fail because the job already exists.

This commit sets the `` to `before-hook-creation,hook-succeeded`
so helm will always delete the jobs before creating them again.

(cherry picked from commit a697c588c43450a9b69e607b052c17a27bd5ac0e)

17 months agoUser-friendly output of Breeze and CI scripts (#12735)
Jarek Potiuk [Tue, 1 Dec 2020 16:44:05 +0000 (17:44 +0100)] 
User-friendly output of Breeze and CI scripts (#12735)

(cherry picked from commit a02e0f746f85e6ac8786bc1d99bf78994e66b5cf)

17 months agoImprove wording of selective checks comments (#12701)
Jarek Potiuk [Sun, 29 Nov 2020 17:27:09 +0000 (18:27 +0100)] 
Improve wording of selective checks comments (#12701)

(cherry picked from commit 7e426d3f4772232a8a707951b52ccef4b4caffa1)

17 months agoEnable PIP check for both CI and PROD image (#12664)
Jarek Potiuk [Fri, 27 Nov 2020 20:33:50 +0000 (21:33 +0100)] 
Enable PIP check for both CI and PROD image (#12664)

This PR enables PIP check after constraints have been updated
to be stable and 'pip check' compliant in #12636

(cherry picked from commit fa8af2d16551e287673d94a40cfb41e49d685412)

17 months agoRequirements shoudl be removed. Sufficient time passed.
Jarek Potiuk [Sun, 13 Dec 2020 11:32:46 +0000 (12:32 +0100)] 
Requirements shoudl be removed. Sufficient time passed.

17 months agoBugfix: Unable to import Airflow plugins on Python 3.8 (#12859)
Kaxil Naik [Fri, 11 Dec 2020 15:28:13 +0000 (15:28 +0000)] 
Bugfix: Unable to import Airflow plugins on Python 3.8 (#12859)


17 months agoScheduler should acknowledge active runs properly (#13803)
John [Thu, 21 Jan 2021 01:33:26 +0000 (20:33 -0500)] 
Scheduler should acknowledge active runs properly (#13803)

This commit fixes the scheduler to acknowledge active runs properly
when comparing against max_active_runs for unscheduled DAG runs.

17 months agoCreated CustomExecutorsRequireFullPathRule class (#13678)
Dr. Dennis Akpenyi [Wed, 20 Jan 2021 17:44:21 +0000 (18:44 +0100)] 
Created CustomExecutorsRequireFullPathRule class (#13678)

closes: #11053

17 months agoKubernetesExecutor should accept images from executor_config (#13074)
Daniel Imberman [Tue, 12 Jan 2021 16:21:19 +0000 (08:21 -0800)] 
KubernetesExecutor should accept images from executor_config (#13074)


Users should be able to specify custom images in the executor_config. Not being
able to is a regression.

17 months agoUpdate URL for docs (#13561)
Kamil Breguła [Sat, 9 Jan 2021 12:27:33 +0000 (13:27 +0100)] 
Update URL for docs (#13561)

17 months agoAdd upgrade check option to list checks (#13392)
Madison Bowden [Thu, 7 Jan 2021 22:49:18 +0000 (14:49 -0800)] 
Add upgrade check option to list checks (#13392)

18 months agoFix Jinja undefined upgrade check for non-templated objects (#13373)
Madison Bowden [Tue, 29 Dec 2020 23:49:33 +0000 (15:49 -0800)] 
Fix Jinja undefined upgrade check for non-templated objects (#13373)

18 months agoMoved boto3 limitation to snowflake (#13286)
Jarek Potiuk [Wed, 23 Dec 2020 17:52:07 +0000 (18:52 +0100)] 
Moved boto3 limitation to snowflake (#13286)

Fixes #13093

18 months agoFix "airflow db upgrade" to upgrade db as intended (#13267)
John [Wed, 23 Dec 2020 05:53:05 +0000 (00:53 -0500)] 
Fix "airflow db upgrade" to upgrade db as intended (#13267)

This commit points the cli "airflow db upgrade" to the intended target of that command.

18 months agoDo not allow yanked version of Airflow in upgrade-check tool (#13160)
Xiaodong DENG [Fri, 18 Dec 2020 20:37:47 +0000 (21:37 +0100)] 
Do not allow yanked version of Airflow in upgrade-check tool (#13160)