airflow.git
2 months ago[Issue#22846] allow option to encode or not encode UUID when uploading from Cassandra...
Xiao Fu [Fri, 20 May 2022 06:44:32 +0000 (23:44 -0700)] 
[Issue#22846] allow option to encode or not encode UUID when uploading from Cassandra to GCS (#23766)

2 months agoFix regression in ignoring symlinks (#23535)
Ian Buss [Fri, 20 May 2022 06:35:41 +0000 (07:35 +0100)] 
Fix regression in ignoring symlinks (#23535)

2 months agoGrid data: do not load all mapped instances (#23813)
Brent Bovenzi [Fri, 20 May 2022 04:18:17 +0000 (00:18 -0400)] 
Grid data: do not load all mapped instances (#23813)

* only get necessary task instances

* add comment

* encode_ti -> get_task_summary

2 months agoFix error when SnowflakeHook take empty list in `sql` param (#23767)
Dmytro Kazanzhy [Fri, 20 May 2022 03:59:25 +0000 (06:59 +0300)] 
Fix error when SnowflakeHook take empty list in `sql` param (#23767)

2 months agoReformat the whole AWS documentation (#23810)
Vincent [Thu, 19 May 2022 23:40:19 +0000 (19:40 -0400)] 
Reformat the whole AWS documentation (#23810)

2 months agoupdate CloudSqlInstanceImportOperator to CloudSQLImportInstanceOperator (#23800)
ishiis [Thu, 19 May 2022 23:02:22 +0000 (08:02 +0900)] 
update CloudSqlInstanceImportOperator to CloudSQLImportInstanceOperator (#23800)

2 months agoFix python version command (#23818)
Jarek Potiuk [Thu, 19 May 2022 22:40:02 +0000 (00:40 +0200)] 
Fix python version command (#23818)

2 months agoUnbreak main after missing classes were added (#23819)
Jarek Potiuk [Thu, 19 May 2022 21:53:31 +0000 (23:53 +0200)] 
Unbreak main after missing classes were added (#23819)

2 months agoCloud SQL assets & system tests migration (AIP-47) (#23583)
Wojciech Januszek [Thu, 19 May 2022 19:17:41 +0000 (21:17 +0200)] 
Cloud SQL assets & system tests migration (AIP-47) (#23583)

2 months agoSupport host_name on Datadog provider (#23784)
Gonzalo Peci [Thu, 19 May 2022 17:20:48 +0000 (19:20 +0200)] 
Support host_name on Datadog provider (#23784)

This is required to use other Datadog tenants like app.datadoghq.eu

2 months agopostgres_operator_howto_guide.rst (#23789)
Ryan Hatter [Thu, 19 May 2022 17:16:24 +0000 (12:16 -0500)] 
postgres_operator_howto_guide.rst (#23789)

Saying "**the** PostgreSQL database" confused me. I thought it was implying that a user could/should connect to the airflow metadata db

2 months agoUpdate the DMS Sample DAG and Docs (#23681)
D. Ferruzzi [Thu, 19 May 2022 16:49:30 +0000 (09:49 -0700)] 
Update the DMS Sample DAG and Docs (#23681)

2 months agoModify db clean to also catch the ProgrammingError exception (#23699)
Jian Yuan Lee [Thu, 19 May 2022 16:47:56 +0000 (17:47 +0100)] 
Modify db clean to also catch the ProgrammingError exception (#23699)

2 months agoFurther speed up fixing ownership in CI (#23782)
Jarek Potiuk [Thu, 19 May 2022 15:43:26 +0000 (17:43 +0200)] 
Further speed up fixing ownership in CI (#23782)

After #23775 I noticed that there is yet another small improvement
area in the CI buld speed. Currently build-ci-image builds and push
only "commit-tagged" images, but "fix-ownership" requires
the "latest" image to run.

This PR adds --tag-as-latest option also to build-image and
build-prod-image commands - similarly as for the pull-image and
pull-prod-image. This will retag the "commit" images as latest in the
build-ci-images step and allow to save 1m on pulling the latest image
before fix-ownership (bringing it back to 1s overhead)

2 months agoAdd `dttm` searchable field in audit log (#23794)
humit [Thu, 19 May 2022 14:09:51 +0000 (23:09 +0900)] 
Add `dttm` searchable field in audit log (#23794)

2 months agoFix python version used for cache preparaation (#23785)
Jarek Potiuk [Thu, 19 May 2022 06:47:47 +0000 (08:47 +0200)] 
Fix python version used for cache preparaation (#23785)

Cache preparation on CI used default (Python 3.7) version of the
image. It had an influence on time of "full build needed" only and
for users who wanted to build breeze image for Python version
different than default Python 3.7.

It had no big influence on "main" builds" because in main we are
build images with "upgrade-to-newer-dependencies" which takes
longer anyway.

2 months agoFix ``SqliteHook`` compatibility with SQLAlchemy engine (#23790)
Kaxil Naik [Thu, 19 May 2022 02:21:32 +0000 (03:21 +0100)] 
Fix ``SqliteHook`` compatibility with SQLAlchemy engine (#23790)

Same as https://github.com/apache/airflow/pull/19508 but for Sqlite as described in https://docs.sqlalchemy.org/en/14/dialects/sqlite.html#connect-strings to be able to create a Sqlalchemy engine from the URI itself.

Without this, it currently fails with the following error due to how we create URI in Connections. An absolute path is denoted by starting with a slash, means you need four slashes:

```
url = sqlite://%2Ftmp%2Fsqlite.db

    def create_connect_args(self, url):
        if url.username or url.password or url.host or url.port:
>           raise exc.ArgumentError(
                "Invalid SQLite URL: %s\n"
                "Valid SQLite URL forms are:\n"
                " sqlite:///:memory: (or, sqlite://)\n"
                " sqlite:///relative/path/to/file.db\n"
                " sqlite:////absolute/path/to/file.db" % (url,)
            )
E           sqlalchemy.exc.ArgumentError: Invalid SQLite URL: sqlite://%2Ftmp%2Fsqlite.db
E           Valid SQLite URL forms are:
E            sqlite:///:memory: (or, sqlite://)
E            sqlite:///relative/path/to/file.db
E            sqlite:////absolute/path/to/file.db
```

2 months agoMaintain grid view selection on filtering upstream (#23779)
Brent Bovenzi [Thu, 19 May 2022 01:52:02 +0000 (21:52 -0400)] 
Maintain grid view selection on filtering upstream (#23779)

* Maintain grid selection on filter upstream

The grid view selection was being cleared when clicking "Filter Upstream". The selection should persist.

Also, added a left margin to the "Reset root" button

* fix linting

2 months agocleanup usage of `get_connections()`` from test suite (#23757)
eladkal [Wed, 18 May 2022 23:42:23 +0000 (02:42 +0300)] 
cleanup usage of `get_connections()`` from test suite (#23757)

The function is deprecated and raises warnings https://github.com/apache/airflow/pull/10192
Replacing the usage with `get_connection()`

2 months agoFallback to parse dag_file when no dag in the db (#23738)
Ping Zhang [Wed, 18 May 2022 23:08:14 +0000 (16:08 -0700)] 
Fallback to parse dag_file when no dag in the db (#23738)

2 months agoclearer method name in scheduler_job.py (#23702)
Ryan Hatter [Wed, 18 May 2022 23:06:16 +0000 (18:06 -0500)] 
clearer method name in scheduler_job.py (#23702)

2 months agoFix auto upstream dep when expanding non-templated field (#23771)
Jed Cunningham [Wed, 18 May 2022 19:43:16 +0000 (13:43 -0600)] 
Fix auto upstream dep when expanding non-templated field (#23771)

If you tried to expand via xcom into a non-templated field without
explicitly setting the upstream task dependency, the scheduler would
crash because the upstream task dependency wasn't being set
automatically. It was being set only for templated fields, but now we do
it for both.

2 months agoAdd tagging image as latest for CI image wait (#23775)
Jarek Potiuk [Wed, 18 May 2022 14:21:47 +0000 (16:21 +0200)] 
Add tagging image as latest for CI image wait (#23775)

The "wait for image" step lacked --tag-as-latest which made the
subsequent "fix-ownership" step run sometimes far longer than
needed - because it rebuilt the image for fix-ownership case.

Also the "fix-ownership" command has been changed to just pull
the image if one is missing locally rather than build. This
command might be run in an environment where the image is missing
or any other image was build (for example in jobs where an image
was build for different Python version) in this case the command
will simply use whatever Python version is available (it does
not matter), or in case no image is available, it will pull the image
as the last resort.

2 months agoChange `Github` to `GitHub` (#23764) 23763/head
John Bampton [Wed, 18 May 2022 08:16:33 +0000 (18:16 +1000)] 
Change `Github` to `GitHub` (#23764)

2 months agoFix doc description of [core] parallelism config setting (#23768)
Chris Redekop [Wed, 18 May 2022 07:08:33 +0000 (01:08 -0600)] 
Fix doc description of [core] parallelism config setting (#23768)

2 months agoHelm chart 1.6.0rc2 (#23754)
Jed Cunningham [Tue, 17 May 2022 17:42:13 +0000 (11:42 -0600)] 
Helm chart 1.6.0rc2 (#23754)

2 months agoFix task log is not captured (#23684)
Ping Zhang [Tue, 17 May 2022 16:29:52 +0000 (09:29 -0700)] 
Fix task log is not captured (#23684)

when StandardTaskRunner runs tasks with exec

Issue: https://github.com/apache/airflow/issues/23540

2 months agoIncrease timeout for Helm Chart executor upgrade tests (#23759)
Jarek Potiuk [Tue, 17 May 2022 16:26:30 +0000 (18:26 +0200)] 
Increase timeout for Helm Chart executor upgrade tests (#23759)

2 months agoAllow more parameters to be piped through via execute_in_subprocess (#23286)
Charles Machalow [Tue, 17 May 2022 15:47:19 +0000 (08:47 -0700)] 
Allow more parameters to be piped through via execute_in_subprocess (#23286)

2 months agoFix broken main branch (#23751)
eladkal [Tue, 17 May 2022 13:43:53 +0000 (16:43 +0300)] 
Fix broken main branch (#23751)

main branch is broken since https://github.com/apache/airflow/pull/23630 needed rebase before merge
as https://github.com/apache/airflow/pull/23730 added the missing example dag

2 months agoUpdated MongoDB logo (#23746)
Ross Lawley [Tue, 17 May 2022 12:29:48 +0000 (13:29 +0100)] 
Updated MongoDB logo (#23746)

As per https://www.mongodb.com/brand-resources

2 months agoAdd 'reschedule' to the serialized fields for the BaseSensorOperator (#23674)
David Caron [Tue, 17 May 2022 12:18:29 +0000 (08:18 -0400)] 
Add 'reschedule' to the serialized fields for the BaseSensorOperator (#23674)

fix #23411

2 months agoSpeech To Text assets & system tests migration (AIP-47) (#23643)
Wojciech Januszek [Tue, 17 May 2022 10:42:54 +0000 (12:42 +0200)] 
Speech To Text assets & system tests migration (AIP-47) (#23643)

Co-authored-by: Wojciech Januszek <januszek@google.com>
2 months agoAdd AWS project structure tests (re: AIP-47) (#23630)
Niko [Tue, 17 May 2022 09:22:49 +0000 (02:22 -0700)] 
Add AWS project structure tests (re: AIP-47) (#23630)

2 months agoDisable flower in chart by default (#23737)
Jed Cunningham [Tue, 17 May 2022 05:03:28 +0000 (23:03 -0600)] 
Disable flower in chart by default (#23737)

2 months agoRemove titles from link buttons (#23736)
Brent Bovenzi [Tue, 17 May 2022 02:58:56 +0000 (22:58 -0400)] 
Remove titles from link buttons (#23736)

2 months agoAdd typing for airflow/configuration.py (#23716)
Jarek Potiuk [Mon, 16 May 2022 21:19:35 +0000 (23:19 +0200)] 
Add typing for airflow/configuration.py (#23716)

* Add typing for airflow/configuration.py

The configuraiton.py did not have typing information and it made
it rather difficult to reason about it-especially that it went
a few changes in the past that made it rather complex to
understand.

This PR adds typing information all over the configuration file

2 months agoAdd version to migration prefix (#23564)
Daniel Standish [Mon, 16 May 2022 21:13:53 +0000 (14:13 -0700)] 
Add version to migration prefix (#23564)

We don't really need the alembic revision id in the filename.  having version instead is much more useful.  having both of them takes up too much space.

2 months agoFix grid details header text overlap (#23728)
Brent Bovenzi [Mon, 16 May 2022 19:21:54 +0000 (15:21 -0400)] 
Fix grid details header text overlap (#23728)

Move top margin to each breadcrumb component to make sure that there is no overlap when the header wraps with long names.

2 months agoAdd doc and sample dag for GCSToS3Operator (#23730)
Vincent [Mon, 16 May 2022 19:16:16 +0000 (15:16 -0400)] 
Add doc and sample dag for GCSToS3Operator (#23730)

2 months agoAdd fields to CLOUD_SQL_EXPORT_VALIDATION. (#23724)
ishiis [Mon, 16 May 2022 19:16:09 +0000 (04:16 +0900)] 
Add fields to CLOUD_SQL_EXPORT_VALIDATION. (#23724)

2 months agoHandle invalid date parsing in webserver views. (#23161)
Karthikeyan Singaravelan [Mon, 16 May 2022 19:15:56 +0000 (00:45 +0530)] 
Handle invalid date parsing in webserver views. (#23161)

* Handle invalid date from query parameters in views.

* Add tests.

* Use common parsing helper.

* Add type hint.

* Remove unwanted error check.

* Fix extra_links endpoint.

2 months agoEnsure execution_timeout as timedelta (#23655)
Ping Zhang [Mon, 16 May 2022 18:53:58 +0000 (11:53 -0700)] 
Ensure execution_timeout as timedelta (#23655)

2 months agoUse profiles to disable flower in docker-compose (#23709)
Kamil Breguła [Mon, 16 May 2022 12:04:41 +0000 (14:04 +0200)] 
Use profiles to disable flower in docker-compose (#23709)

3 months agoAdd slim image to docs/docker-stack/README.md (#23710)
Kamil Breguła [Sun, 15 May 2022 09:52:51 +0000 (11:52 +0200)] 
Add slim image to docs/docker-stack/README.md (#23710)

3 months agoAdd UI tests for /utils and /components (#23456)
Brent Bovenzi [Fri, 13 May 2022 14:58:28 +0000 (10:58 -0400)] 
Add UI tests for /utils and /components (#23456)

* Add UI tests for /utils and /components

* add test for Table

* Address PR feedback

* Fix window prompt var

* Fix TaskName test from rebase

* fix lint errors

3 months agoAdd environment check and build image check for more Breeze commands (#23687)
Jarek Potiuk [Fri, 13 May 2022 13:16:33 +0000 (15:16 +0200)] 
Add environment check and build image check for more Breeze commands (#23687)

Several commands of Breeze depends on docker, docker compose
being available as well as breeze image. They will work
fine if you "just" built the image but they might benefit
from the image being rebuilt (to make sure all latest
dependencies are installed in the image). The common checks
done in "shell" command for that are now extracted to common
utils and run as first thing in those commands that need it.

3 months agoClarify that bundle extras should not be used for PyPi installs (#23697)
Jarek Potiuk [Fri, 13 May 2022 11:33:17 +0000 (13:33 +0200)] 
Clarify that bundle extras should not be used for PyPi installs (#23697)

The bundle extras we have are only used for development and they
should not be used to install airflow from PyPI. This update
to documentation clarifies it.

Closes: #23692

3 months agoFix property name in breeze Shell Params (#23696)
Jarek Potiuk [Fri, 13 May 2022 11:00:39 +0000 (13:00 +0200)] 
Fix property name in breeze Shell Params (#23696)

The rename from #23562 missed few shell_parms usage where it
also should be replaced.

3 months agoDisable Flower by default from docker-compose (#23685)
Jarek Potiuk [Fri, 13 May 2022 10:21:36 +0000 (12:21 +0200)] 
Disable Flower by default from docker-compose (#23685)

3 months agoAdd git_source to DatabricksSubmitRunOperator (#23620)
akolar-db [Fri, 13 May 2022 09:56:13 +0000 (11:56 +0200)] 
Add git_source to DatabricksSubmitRunOperator (#23620)

The existing `DatabricksSubmitRunOperator` is extended with the support for the `git_source` parameter which allows users to run notebook tasks from files committed to git repositories.

If specified, any notebook task that is part of the payload will clone the repository and check out the commit, tag, or the tip of the specified branch. This is an alternative to dev repos ([docs](https://docs.databricks.com/repos/index.html)) where the checkout/update would have to be triggered manually.

Public documentation for the feature available here: https://docs.databricks.com/dev-tools/api/latest/jobs.html (NB: as noted in the docs, the feature is currently in public preview).

3 months agoUse func.count to count rows (#23657)
Ping Zhang [Thu, 12 May 2022 21:49:06 +0000 (14:49 -0700)] 
Use func.count to count rows (#23657)

3 months agoUpdate doc and sample dag for Quicksight (#23653)
Vincent [Thu, 12 May 2022 20:19:35 +0000 (14:19 -0600)] 
Update doc and sample dag for Quicksight (#23653)

3 months agoFix expand/collapse all buttons (#23590)
Brent Bovenzi [Thu, 12 May 2022 19:48:31 +0000 (15:48 -0400)] 
Fix expand/collapse all buttons (#23590)

* communicate via customevents

* Handle open group logic in wrapper

* fix tests

* Make grid action buttons sticky

* Add default toggle fn

* fix splitting task id by '.'

* fix missing dagrun ids

3 months agoMove around overflow, position and padding (#23044)
Brent Bovenzi [Thu, 12 May 2022 19:47:24 +0000 (15:47 -0400)] 
Move around overflow, position and padding (#23044)

3 months agoremove stale serialized dags (#22917)
Ping Zhang [Thu, 12 May 2022 19:01:47 +0000 (12:01 -0700)] 
remove stale serialized dags (#22917)

3 months agoShorten max pre-commit hook name length (#23677)
Daniel Standish [Thu, 12 May 2022 18:46:56 +0000 (11:46 -0700)] 
Shorten max pre-commit hook name length (#23677)

When names are too long, pre-commit output looks very ugly and takes up 2x lines. Here I reduce max length just a little bit further so that pre-commit output renders properly on a macbook pro 16" with terminal window splitting screen horizontally.

3 months agoUpgrade `pip` to latest released 22.1.0 version (#23665)
Jarek Potiuk [Thu, 12 May 2022 17:36:06 +0000 (19:36 +0200)] 
Upgrade `pip` to latest released 22.1.0 version (#23665)

We are finally able to get rid of the annoying false-positive
warnings and we have finally a chance on having warning-free
installation during docker builds.

3 months agoReplace "absolute()" with "resolve()" in pathlib objects (#23675)
Jarek Potiuk [Thu, 12 May 2022 17:30:39 +0000 (19:30 +0200)] 
Replace "absolute()" with "resolve()" in pathlib objects (#23675)

TIL that absolute() is an undocumented in Pathlib and that we
should use resolve() instead.

So this is it.

3 months agoAdd wildcard possibility to `package-filter` parametere (#23672)
Jarek Potiuk [Thu, 12 May 2022 17:23:38 +0000 (19:23 +0200)] 
Add wildcard possibility to `package-filter` parametere (#23672)

the glob parameters (for example `apache-airflow-providers-*`) did
not work because only fixed list of parameters was allowed.

This PR converts the package-filter parameter to stop verifying the
value passed - so autocomplete continues to work but you should
still be able to use glob.

It also removes few places where the parameters were used with
`--` separator.

3 months agoMigrate Dataproc to new system tests design (#22777)
Bartłomiej Hirsz [Thu, 12 May 2022 16:49:10 +0000 (18:49 +0200)] 
Migrate Dataproc to new system tests design (#22777)

3 months agoSynchronize support for Postgres and K8S in docs (#23673)
Jarek Potiuk [Thu, 12 May 2022 13:26:04 +0000 (15:26 +0200)] 
Synchronize support for Postgres and K8S in docs (#23673)

We just added support for Postgres 14 and K8S 1.24 and since we
did not have any changes to support either in main we are bringing
the support to 2.3 line as well.

This documentation syncs all remaining places where it should be
updated.

3 months agoremove `--` in `./breeze build-docs` command (#23671)
ishiis [Thu, 12 May 2022 12:33:53 +0000 (21:33 +0900)] 
remove `--` in `./breeze build-docs` command (#23671)

3 months agoAIP45 Remove dag parsing in airflow run local (#21877)
Ping Zhang [Thu, 12 May 2022 10:09:48 +0000 (03:09 -0700)] 
AIP45 Remove dag parsing in airflow run local (#21877)

3 months agoPrepare provider documentation 2022.05.11 (#23631) providers-google/7 providers-amazon/3.4.0 providers-amazon/3.4.0rc1 providers-apache-beam/3.4.0 providers-apache-beam/3.4.0rc1 providers-apache-hive/2.3.3 providers-apache-hive/2.3.3rc1 providers-cncf-kubernetes/4.0.2 providers-cncf-kubernetes/4.0.2rc1 providers-databricks/2.7.0 providers-databricks/2.7.0rc1 providers-docker/2.7.0 providers-docker/2.7.0rc1 providers-google/7.0.0 providers-google/7.0.0rc1 providers-jira/2.0.5 providers-jira/2.0.5rc1 providers-microsoft-azure/3.9.0 providers-microsoft-azure/3.9.0rc1 providers-opsgenie/3.1.0 providers-opsgenie/3.1.0rc1 providers-presto/2.2.1 providers-presto/2.2.1rc1 providers-salesforce/3.4.4 providers-salesforce/3.4.4rc1 providers-snowflake/2.7.0 providers-snowflake/2.7.0rc1 providers-ssh/2.4.4 providers-ssh/2.4.4rc1 providers-tableau/2.1.8 providers-tableau/2.1.8rc1 providers-trino/2.3.0 providers-trino/2.3.0rc1
Jarek Potiuk [Wed, 11 May 2022 23:01:16 +0000 (01:01 +0200)] 
Prepare provider documentation 2022.05.11 (#23631)

Co-authored-by: eladkal <45845474+eladkal@users.noreply.github.com>
Co-authored-by: eladkal <45845474+eladkal@users.noreply.github.com>
3 months agoRevert "Fix k8s pod.execute randomly stuck indefinitely by logs consumption (#23497...
Jarek Potiuk [Wed, 11 May 2022 22:20:02 +0000 (00:20 +0200)] 
Revert "Fix k8s pod.execute randomly stuck indefinitely by logs consumption (#23497) (#23618)" (#23656)

This reverts commit ee342b85b97649e2e29fcf83f439279b68f1b4d4.

3 months agoRename cluster_policy to task_policy (#23468)
humit [Wed, 11 May 2022 19:40:10 +0000 (04:40 +0900)] 
Rename cluster_policy to task_policy (#23468)

* Rename cluster_policy to task_policy

* rename task_policy as example_task_policy.

3 months ago[FEATURE] google provider - BigQueryInsertJobOperator log query (#23648)
raphaelauv [Wed, 11 May 2022 19:28:19 +0000 (21:28 +0200)] 
[FEATURE] google provider - BigQueryInsertJobOperator log query (#23648)

3 months agoFix k8s pod.execute randomly stuck indefinitely by logs consumption (#23497) (#23618)
Sebastian Chamena [Wed, 11 May 2022 19:20:49 +0000 (12:20 -0700)] 
Fix k8s pod.execute randomly stuck indefinitely by logs consumption (#23497) (#23618)

3 months agoFixed test and remove pytest.mark.xfail for test_exc_tb (#23650)
Kanthi [Wed, 11 May 2022 19:16:49 +0000 (15:16 -0400)] 
Fixed test and remove pytest.mark.xfail for test_exc_tb (#23650)

3 months agoAdded kubernetes version (1.24) in README.md(for Main version(dev)), … (#23649)
Kanthi [Wed, 11 May 2022 17:13:01 +0000 (13:13 -0400)] 
Added kubernetes version (1.24) in README.md(for Main version(dev)), … (#23649)

* Added kubernetes version (1.24) in README.md(for Main version(dev)), accidentally removed in merge cnflict.

* Update README.md

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
3 months agoAdd `RedshiftDeleteClusterOperator` support (#23563)
pankajastro [Wed, 11 May 2022 17:07:01 +0000 (22:37 +0530)] 
Add `RedshiftDeleteClusterOperator` support (#23563)

Add support for `RedshiftDeleteClusterOperator`. This will help to clean resources using airflow operators when needed. In the current implementation, By default, I'm waiting until the cluster is completely removed to return immediately without waiting set `wait_for_completion` param to False

- Add operator class
- Add basic unit test
- Add an example task
- Add relevant documentation

3 months agoAdded postgres 14 to support versions(including breeze) (#23506)
Kanthi [Wed, 11 May 2022 16:26:19 +0000 (12:26 -0400)] 
Added postgres 14 to support versions(including breeze) (#23506)

* Added postgres 14 to support versions(including breeze)

3 months agoDon't run pre-migration checks for downgrade (#23634)
Daniel Standish [Wed, 11 May 2022 16:08:06 +0000 (09:08 -0700)] 
Don't run pre-migration checks for downgrade (#23634)

These checks are only make sense for upgrades.  Generally they exist to resolve referential integrity issues etc before adding constraints.  In the downgrade context, we generally only remove constraints, so it's a non-issue.

3 months agoAdd index for event column in log table (#23625)
Gabriel Machado [Wed, 11 May 2022 14:45:33 +0000 (16:45 +0200)] 
Add index for event column in log table (#23625)

3 months agoSimplify flash message for _airflow_moved tables (#23635)
Daniel Standish [Wed, 11 May 2022 14:13:57 +0000 (07:13 -0700)] 
Simplify flash message for _airflow_moved tables (#23635)

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
3 months agoFix assuming "Feature" answer on CI when generating docs (#23640)
Jarek Potiuk [Wed, 11 May 2022 11:15:22 +0000 (13:15 +0200)] 
Fix assuming "Feature" answer on CI when generating docs (#23640)

We have now different answers posisble when generating docs, and
for testing we assume we answered randomly during the generation
of documentation.

3 months agoFix typo issue (#23633)
humit [Wed, 11 May 2022 10:58:26 +0000 (19:58 +0900)] 
Fix typo issue (#23633)

3 months ago[FEATURE] add K8S 1.24 support (#23637)
raphaelauv [Wed, 11 May 2022 10:52:24 +0000 (12:52 +0200)] 
[FEATURE] add K8S 1.24 support (#23637)

3 months ago[FEATURE] update K8S-KIND to 0.13.0 (#23636)
raphaelauv [Wed, 11 May 2022 08:26:14 +0000 (10:26 +0200)] 
[FEATURE] update K8S-KIND to 0.13.0 (#23636)

3 months agoPrevent KubernetesJobWatcher getting stuck on resource too old (#23521)
Ruben Laguna [Wed, 11 May 2022 06:25:49 +0000 (08:25 +0200)] 
Prevent KubernetesJobWatcher getting stuck on resource too old (#23521)

* Prevent KubernetesJobWatcher getting stuck on resource too old

If the watch fails because "resource too old" the
KubernetesJobWatcher should not retry with the same resource version
as that will end up in loop where there is no progress.

* Reset ResourceVersion().resource_version to 0

3 months agoMake provider doc preparation a bit more fun :) (#23629)
Jarek Potiuk [Tue, 10 May 2022 22:19:54 +0000 (00:19 +0200)] 
Make provider doc preparation a bit more fun :) (#23629)

Previously you had to manually add versions when changelog was
modified. But why not to get a bit more fun and get the versions
bumped automatically based on your assesment when reviewing the
provideers rather than after looking at the generated changelog.

3 months agoFix: Exception when parsing log #20966 (#23301)
Jakub Novák [Tue, 10 May 2022 20:43:25 +0000 (22:43 +0200)] 
Fix: Exception when parsing log #20966 (#23301)

* UnicodeDecodeError: 'utf-8' codec can't decode byte 0xXX in position X: invalid start byte

  File "/opt/work/python395/lib/python3.9/site-packages/airflow/hooks/subprocess.py", line 89, in run_command
    line = raw_line.decode(output_encoding).rstrip()            # raw_line ==  b'\x00\x00\x00\x11\xa9\x01\n'
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 4: invalid start byte

* Update subprocess.py

* Update subprocess.py

* Fix:  Exception when parsing log #20966

* Fix:  Exception when parsing log #20966

 Another alternative is: try-catch it.

e.g.

```
            line = ''
            for raw_line in iter(self.sub_process.stdout.readline, b''):
                try:
                    line = raw_line.decode(output_encoding).rstrip()
                except UnicodeDecodeError as err:
                    print(err, output_encoding, raw_line)
                self.log.info("%s", line)
```

* Create test_subprocess.sh

* Update test_subprocess.py

* Added shell directive and license to test_subprocess.sh

* Distinguish between raw and decoded lines as suggested by @uranusjr

* simplify test

Co-authored-by: muhua <microhuang@live.com>
3 months agoImplement send_callback method for CeleryKubernetesExecutor and LocalKubernetesExecut...
mhenc [Tue, 10 May 2022 17:13:00 +0000 (19:13 +0200)] 
Implement send_callback method for CeleryKubernetesExecutor and LocalKubernetesExecutor (#23617)

3 months ago[FEATURE] google provider - split GkeStartPodOperator execute (#23518)
raphaelauv [Tue, 10 May 2022 15:51:37 +0000 (17:51 +0200)] 
[FEATURE] google provider - split GkeStartPodOperator execute (#23518)

3 months agoFixed Kubernetes Operator large xcom content Defect (#23490)
rahulgoyal2987 [Tue, 10 May 2022 15:46:55 +0000 (21:16 +0530)] 
Fixed Kubernetes Operator large xcom content Defect  (#23490)

3 months agoAdd slim images to docker-stack docs index (#23601)
Jarek Potiuk [Tue, 10 May 2022 15:24:26 +0000 (17:24 +0200)] 
Add slim images to docker-stack docs index (#23601)

3 months agoAdd Quicksight create ingestion Hook and Operator (#21863)
Harpreet Singh [Tue, 10 May 2022 14:54:13 +0000 (20:24 +0530)] 
Add Quicksight create ingestion Hook and Operator (#21863)

* Add Quicksight create ingestion Hook and Operator

Co-authored-by: eladkal <45845474+eladkal@users.noreply.github.com>
3 months agoMake Breeze help generation indepdent from having breeze installed (#23612)
Jarek Potiuk [Tue, 10 May 2022 09:49:39 +0000 (11:49 +0200)] 
Make Breeze help generation indepdent from having breeze installed (#23612)

Generation of Breeze help requires breeze to be installed. However
if you have locally installed breeze with different dependencies
and did not run self-upgrade, the results of generation of the
images might be different (for example when different rich
version is used). This change works in the way that:
* you do not have to have breeze installed at all to make it work
* it always upgrades to latest breeze when it is not installed
* but this only happens when you actually modified some breeze code

3 months agoAdd exportContext.offload flag to CLOUD_SQL_EXPORT_VALIDATION. (#23614)
ishiis [Tue, 10 May 2022 09:49:18 +0000 (18:49 +0900)] 
Add exportContext.offload flag to CLOUD_SQL_EXPORT_VALIDATION. (#23614)

3 months agoUpdate min requirements for rich to 12.4.1 (#23604)
Jarek Potiuk [Tue, 10 May 2022 06:36:28 +0000 (08:36 +0200)] 
Update min requirements for rich to 12.4.1 (#23604)

3 months agoAdd sample dag and doc for S3ListPrefixesOperator (#23448)
Vincent [Mon, 9 May 2022 22:54:59 +0000 (16:54 -0600)] 
Add sample dag and doc for S3ListPrefixesOperator (#23448)

* Add sample dag and doc for S3ListPrefixesOperator

* Fix static checks

3 months agoAdd exception to catch single line private keys (#23043)
nsAstro [Mon, 9 May 2022 22:49:22 +0000 (18:49 -0400)] 
Add exception to catch single line private keys (#23043)

3 months agoUse inclusive words in apache airflow project (#23090)
Edith Puclla [Mon, 9 May 2022 21:52:29 +0000 (16:52 -0500)] 
Use inclusive words in apache airflow project (#23090)

3 months agoImprove caching for multi-platform images. (#23562)
Jarek Potiuk [Mon, 9 May 2022 21:02:25 +0000 (23:02 +0200)] 
Improve caching for multi-platform images. (#23562)

This is another attempt to improve caching performance for
multi-platform images as the previous ones were undermined by a
bug in buildx multiplatform cache-to implementattion that caused
the image cache to be overwritten between platforms,
when multiple images were build.

The bug is created for the buildx behaviour at
https://github.com/docker/buildx/issues/1044 and until it is fixed
we have to prpare separate caches for each platform and push them
to separate tags.

That adds a bit overhead on the building step, but for now it is
the simplest way we can workaround the bug if we do not want to
manually manipulate manifests and images.

3 months ago19943 Grid view status filters (#23392)
pierrejeambrun [Mon, 9 May 2022 20:32:02 +0000 (22:32 +0200)] 
19943 Grid view status filters (#23392)

* Move tree filtering inside react and add some filters

* Move filters from context to utils

* Fix tests for useTreeData

* Fix last tests.

* Add tests for useFilters

* Refact to use existing SimpleStatus component

* Additional fix after rebase.

* Update following bbovenzi code review

* Update following code review

* Fix tests.

* Fix page flickering issues from react-query

* Fix side panel and small changes.

* Use default_dag_run_display_number in the filter options

* Handle timezone

* Fix flaky test

Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>
3 months agoAdd sample dag and doc for S3ListOperator (#23449)
Vincent [Mon, 9 May 2022 18:21:51 +0000 (12:21 -0600)] 
Add sample dag and doc for S3ListOperator (#23449)

* Add sample dag and doc for S3ListOperator

* Fix doc

3 months agoHelm chart 1.6.0rc1 (#23548)
Jed Cunningham [Mon, 9 May 2022 18:14:44 +0000 (12:14 -0600)] 
Helm chart 1.6.0rc1 (#23548)

3 months agoAdd doc and sample dag for EC2 (#23547)
Vincent [Mon, 9 May 2022 17:56:50 +0000 (11:56 -0600)] 
Add doc and sample dag for EC2 (#23547)

3 months agoApply specific ID collation to root_dag_id too (#23536)
Michael Peteuil [Mon, 9 May 2022 17:48:11 +0000 (13:48 -0400)] 
Apply specific ID collation to root_dag_id too (#23536)

In certain databases there is a need to set the collation for ID fields
like dag_id or task_id to something different than the database default.
This is because in MySQL with utf8mb4 the index size becomes too big for
the MySQL limits. In past pull requests this was handled
[#7570](https://github.com/apache/airflow/pull/7570),
[#17729](https://github.com/apache/airflow/pull/17729), but the
root_dag_id field on the dag model was missed. Since this field is used
to join with the dag_id in various other models ([and
self-referentially](https://github.com/apache/airflow/blob/451c7cbc42a83a180c4362693508ed33dd1d1dab/airflow/models/dag.py#L2766)),
it also needs to have the same collation as other ID fields.

This can be seen by running `airflow db reset` before and after applying
this change while also specifying `sql_engine_collation_for_ids` in the
configuration.

Other related PRs
[#19408](https://github.com/apache/airflow/pull/19408)