airflow.git
4 min agoMerge-friendly output of command hashes for breeze (#24711) main
Jarek Potiuk [Tue, 28 Jun 2022 21:58:01 +0000 (23:58 +0200)] 
Merge-friendly output of command hashes for breeze (#24711)

We had just one hash generated from all commands in breeze and that
basically meant that when there were two PRs on two different
commands in Breeze, they resulted with merge conflict which
should be solved with `breeze regenerate-command-images`.

This change turns the hash output into a multi-command one - i.e.
each command has its own hash, which will make it much more
merge-friendly - i.e. if two PRs will work on two different commands
the rebase should result with merge rather than conflict.

31 min agoScript to filter candidates for PR of the month based on heuristics (#24654)
Jarek Potiuk [Tue, 28 Jun 2022 21:30:59 +0000 (23:30 +0200)] 
Script to filter candidates for PR of the month based on heuristics (#24654)

This scripts proposes top candidates for PR of the month
based on simple heuristics as discussed in the document

https://docs.google.com/document/d/1qO5FztgzJLccfvbagX8DLh1EwhFVD2nUqbw96fRJmQQ/edit?disco=AAAAZ-Ct0Bs&usp_dm=true

39 min agoFix unnecessary check for ARM images (#24718)
Jarek Potiuk [Tue, 28 Jun 2022 21:22:23 +0000 (23:22 +0200)] 
Fix unnecessary check for ARM images (#24718)

The ARM image build introduced in #24664 had problem with build
image that was additionally checking for arm images which were
moved out to a spearate step

2 hours agoAdd AWS operators to create and delete RDS Database (#24099)
Eugene Karimov [Tue, 28 Jun 2022 19:32:17 +0000 (21:32 +0200)] 
Add AWS operators to create and delete RDS Database (#24099)

* Add RdsCreateDbInstanceOperator

* Add RdsDeleteDbInstanceOperator

3 hours agoDebounce status highlighting in Grid view (#24710)
Brent Bovenzi [Tue, 28 Jun 2022 18:50:02 +0000 (14:50 -0400)] 
Debounce status highlighting in Grid view (#24710)

* Add delay / debounce to not always highlight tasks

* fix linting

* single delay variable at 200ms

3 hours agoAdd ARM image building for regular PRs (#24664)
Jarek Potiuk [Tue, 28 Jun 2022 18:20:13 +0000 (20:20 +0200)] 
Add ARM image building for regular PRs (#24664)

The image building for ARM is currently only done in the main build
only to refresh cache, however there are sometimes cases when
new dependency (for example #24635) broke ARM image build and it
was only discovered after merge.

This PR adds extra ARM-based build that should be run after
the AMD64 build. It should not influence the depending steps,
it should just signal failure of the PR if the ARM image cannot
be build.

6 hours agoProtect against using "hook_params" in "get_hook" until we move to 2.3+ (#24706)
Jarek Potiuk [Tue, 28 Jun 2022 16:00:16 +0000 (18:00 +0200)] 
Protect against using "hook_params" in "get_hook" until we move to 2.3+ (#24706)

The "hook_params" method was added in Airflow 2.3 and we should not
use it in providers (yet).

6 hours agofix document about response_check in HttpSensor (#24708)
ishiis [Tue, 28 Jun 2022 16:00:01 +0000 (01:00 +0900)] 
fix document about response_check in HttpSensor (#24708)

6 hours agoRemove `hook-class-names` from provider.yaml (#24702)
eladkal [Tue, 28 Jun 2022 15:46:52 +0000 (18:46 +0300)] 
Remove `hook-class-names` from provider.yaml (#24702)

* Remove `hook-class-names` from provider.yaml
now that providers>=2.2 there is no need for the hook-class-names any longer.
refrence to https://github.com/apache/airflow/pull/17775

* fix `airflow/provider.yaml.schema.json` and `provider_info.schema.json`

* undo changes in `provider_info.schema.json`

6 hours agoFix Grid vertical scrolling (#24684)
Brent Bovenzi [Tue, 28 Jun 2022 15:43:28 +0000 (11:43 -0400)] 
Fix Grid vertical scrolling (#24684)

* fix vertical scrolling

* fix flex grow for panel open/close

* add type checking

* add duration axis component

* remove details/grid width changes

this should be done in a separate PR

6 hours agoAdd `airflow_kpo_in_cluster` label to KPO pods (#24658)
Jed Cunningham [Tue, 28 Jun 2022 15:30:02 +0000 (09:30 -0600)] 
Add `airflow_kpo_in_cluster` label to KPO pods (#24658)

This allows one to determine if the pod was created with in_cluster config
or not, both on the k8s side and in pod_mutation_hooks.

6 hours agodon't try to render child rows for closed groups (#24637)
Brent Bovenzi [Tue, 28 Jun 2022 15:28:27 +0000 (11:28 -0400)] 
don't try to render child rows for closed groups (#24637)

7 hours agoRemove "Label when approved" workflow (#24704)
Jarek Potiuk [Tue, 28 Jun 2022 14:23:02 +0000 (16:23 +0200)] 
Remove "Label when approved" workflow (#24704)

The labelling workflow has proven to be far less useful than we
thought and some of the recent changes in selective checks made
it largely obsolete. The committers can still add "full tests needed"
label when they think it is needed and there is no need to label
the PRs automatically for that (or any other reason).

For quite a while this workflow is basically a useless noise.

9 hours agoMigrate Google example DAG s3_to_gcs to new design AIP-47 (#24641)
Chenglong Yan [Tue, 28 Jun 2022 12:49:47 +0000 (20:49 +0800)] 
Migrate Google example DAG s3_to_gcs to new design AIP-47 (#24641)

related: #22447, #22430

9 hours agoAdd batch option to `SqsSensor` (#24554)
TungHoang [Tue, 28 Jun 2022 12:47:46 +0000 (14:47 +0200)] 
Add batch option to `SqsSensor` (#24554)

Add batch option to `SqsSensor`

Co-authored-by: TungHoang <st.hoang@jellysmack.com>
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
9 hours ago`WebHDFSHook` Bugfix/optional port (#24550)
Ankur Bajaj [Tue, 28 Jun 2022 12:44:09 +0000 (14:44 +0200)] 
`WebHDFSHook` Bugfix/optional port (#24550)

12 hours agoAdd `test_connection` method to Databricks hook (#24617)
Phani Kumar [Tue, 28 Jun 2022 10:01:07 +0000 (15:31 +0530)] 
Add `test_connection` method to Databricks hook (#24617)

This PR enables the test button on the airflow UI for testing whether the Databricks connection works using the connection params filled by the user

12 hours agoClear next method when clearing TIs (#23929)
Tanel Kiis [Tue, 28 Jun 2022 09:40:34 +0000 (12:40 +0300)] 
Clear next method when clearing TIs (#23929)

15 hours agoRename 'resources' arg in Kub op to k8s_resources (#24673)
Tzu-ping Chung [Tue, 28 Jun 2022 06:45:31 +0000 (14:45 +0800)] 
Rename 'resources' arg in Kub op to k8s_resources (#24673)

15 hours agoClean up task decorator type hints and docstrings (#24667)
Tzu-ping Chung [Tue, 28 Jun 2022 06:45:00 +0000 (14:45 +0800)] 
Clean up task decorator type hints and docstrings (#24667)

17 hours agoRemove commented out code (#24685)
Andrey Anshin [Tue, 28 Jun 2022 04:04:19 +0000 (08:04 +0400)] 
Remove commented out code (#24685)

19 hours agoTriggerDagRunOperator.operator_extra_links is attr (#24676)
Tzu-ping Chung [Tue, 28 Jun 2022 02:59:00 +0000 (10:59 +0800)] 
TriggerDagRunOperator.operator_extra_links is attr (#24676)

There's absolutely no reason this needs to be a property. And it cannot
be since we need to access this at the class level.

23 hours agokeep Log scrolling only to nearest div (#24689)
Brent Bovenzi [Mon, 27 Jun 2022 22:53:54 +0000 (18:53 -0400)] 
keep Log scrolling only to nearest div (#24689)

24 hours agoUpdated Snowflake provider connection documentation (#24573)
Daniel Reeves [Mon, 27 Jun 2022 21:58:10 +0000 (17:58 -0400)] 
Updated Snowflake provider connection documentation (#24573)

* Updated Snowflake provider connection documentation

* Moved paragraph down.

* Minor rephrasing to improve clarity

24 hours agoPass worker annotations to generated pod_template_file (#24647)
Gabriel Machado [Mon, 27 Jun 2022 21:57:35 +0000 (23:57 +0200)] 
Pass worker annotations to generated pod_template_file (#24647)

25 hours agoAdd dataset model (#24613)
Daniel Standish [Mon, 27 Jun 2022 20:39:44 +0000 (13:39 -0700)] 
Add dataset model (#24613)

Add model for storing references to datasets, a fundamental component of AIP-48.

25 hours agoAdd Core SQL Provider (#24476)
Benji Lampel [Mon, 27 Jun 2022 20:29:33 +0000 (16:29 -0400)] 
Add Core SQL Provider (#24476)

Adds operators, tests, and new and updated docs for a Core SQL
Provider. This provider is made in favor of adding these operators
to the existing SQL operators in core Airflow. The new provider
will allow for quicker development cycles.

closes: #23874, #24422
related: #23915

35 hours agoTurn inability of creating mypy-cache into warning (#24677)
Jarek Potiuk [Mon, 27 Jun 2022 10:21:14 +0000 (12:21 +0200)] 
Turn inability of creating mypy-cache into warning (#24677)

The mypy cache creation might happen before checking if good
version of docker is installed and it works, so we want to turn
it into a warning rather than error when it is not possible to
create one.

45 hours agoFix behaviour of build/pull after recent Breeze changes (#24657)
Jarek Potiuk [Mon, 27 Jun 2022 01:00:10 +0000 (03:00 +0200)] 
Fix behaviour of build/pull after recent Breeze changes (#24657)

The behaviour of Breeze after some recent changes related to
pulling and building images in parallel have been slightly broken.
Nothing serious but slightly annoying behaviour:

* when starting breeze shell, the image was attempted to be
  build even if it was not needed (but cache efficiency made it fast
  enough to not be too annoying (unless we updated to newer
  python base image

* breeze pull command for "latest" branch makes no sense any more -
  we stopped pushing "latest" image to ghcr.io, we only push
  cache and "tagged" images. We are now turning --image-tag as
  required in "pull_image" and when someone specifies latest,
  error and helpful message is printed

* --force-build flag in "shell-related-commands" was not
  properly propagated to build-image so it did not actually
  force image building.

All those problems are fixed now.

2 days agoS3ToSnowflakeOperator: escape single quote in s3_keys (#24607)
Andrey Anshin [Sun, 26 Jun 2022 15:21:30 +0000 (19:21 +0400)] 
S3ToSnowflakeOperator: escape single quote in s3_keys (#24607)

2 days agoFix handling of GitHub Event types for new selective checks (#24665)
Jarek Potiuk [Sun, 26 Jun 2022 10:11:52 +0000 (12:11 +0200)] 
Fix handling of GitHub Event types for new selective checks (#24665)

One more finding after merging the selective checks in Python
- I missed a case of "pull_request_target".

Fixed and added more tests.

2 days agoHandle "workflow_run" event properly in selective-check (#24656)
Jarek Potiuk [Sun, 26 Jun 2022 09:07:49 +0000 (11:07 +0200)] 
Handle "workflow_run" event properly in selective-check (#24656)

Unfortunately testing workflow_run is a bit difficult because
the changes are only effective after merging them.

Fixing (hopefully) yet another mistake in the workflow run where
commit hash was passed as event name (?)

We are going to handle "workflow_run" as valid event type so this
should now work without passing any event.

3 days agoUse target commit SHA for build image workflow (#24659)
Jarek Potiuk [Sat, 25 Jun 2022 21:34:11 +0000 (23:34 +0200)] 
Use target commit SHA for build image workflow (#24659)

The build-image workflow should use TARGET_COMMIT_SHA as the
selective check COMMIT_REF otherwise it might not build
image when needed

3 days agoChange postgres auth method (#24640)
Gabriel Machado [Sat, 25 Jun 2022 19:23:08 +0000 (21:23 +0200)] 
Change postgres auth method (#24640)

3 days agoMigrate Google example DAG bigquery_transfer to new design AIP-47 (#24543)
Chenglong Yan [Sat, 25 Jun 2022 15:51:26 +0000 (23:51 +0800)] 
Migrate Google example DAG bigquery_transfer to new design AIP-47 (#24543)

related: #22447, #22430

3 days agoRemove selective checks from the "release workflow" (#24655)
Jarek Potiuk [Sat, 25 Jun 2022 15:45:08 +0000 (17:45 +0200)] 
Remove selective checks from the "release workflow" (#24655)

Missed that one too :(

3 days agoRemove misleading message from CI (#24650)
Jarek Potiuk [Sat, 25 Jun 2022 09:27:21 +0000 (11:27 +0200)] 
Remove misleading message from CI (#24650)

The CI jobs do not usually have mypy_cache volume created and
almost all jobs print misleading "mypy_cache_volume" error at
the beginning.

This is a noise - we are not interested in this stderr printed
message - we are only interested in returncode from checking
if it exists.

3 days agoCleanup references to selective checks (#24649)
Jarek Potiuk [Sat, 25 Jun 2022 09:26:52 +0000 (11:26 +0200)] 
Cleanup references to selective checks (#24649)

Selective checks docs have been moved to breeze as part of #24610
but some of the references were still left.

This PR cleans it up.

3 days agoSwitch to new selective-checks in label-when-reviewed workflow (#24651)
Jarek Potiuk [Sat, 25 Jun 2022 09:20:48 +0000 (11:20 +0200)] 
Switch to new selective-checks in label-when-reviewed workflow (#24651)

When #24610 was implemented I missed the label-when-reviewed workflow

3 days agoConvert selective checks to Breeze Python (#24610)
Jarek Potiuk [Sat, 25 Jun 2022 07:46:13 +0000 (09:46 +0200)] 
Convert selective checks to Breeze Python (#24610)

Instead of bash-based, complex logic script to perform PR selective
checks we now integrated the whole logic into Breeze Python code.

It is now much simplified, when it comes to algorithm. We've
implemented simple rule-based decision tree. The rules describing
the decision tree are now are now much easier
to reason about and they correspond one-to-one with the rules
that are implemented in the code in rather straightforward way.

The code is much simpler and diagnostics of the selective checks
has also been vastly improved:

* The rule engine displays status of applying each rule and
  explains (with yellow warning message what decision was made
  and why. Informative messages are printed showing the resulting
  output

* List of files impacting the decision are also displayed

* The names of "ci file group" and "test type" were aligned

* Unit tests covering wide range of cases are added. Each test
  describes what is the case they demonstrate

* `breeze selective-checks` command that is used in CI can also
  be used locally by just providing commit-ish reference of the
  commit to check. This way you can very easily debug problems and
  fix them

Fixes: #19971

3 days agoRemove short version of colour flag (#24639)
Gabriel Machado [Sat, 25 Jun 2022 00:05:32 +0000 (02:05 +0200)] 
Remove short version of colour flag (#24639)

3 days agoAllows to specify different Python base image when building images (#24634)
Jarek Potiuk [Fri, 24 Jun 2022 22:02:57 +0000 (00:02 +0200)] 
Allows to specify different Python base image when building images (#24634)

Today Python released new images that seems to break some of
our dependencies (at least on M1/ARM). This PR adds a workaround
possibility to add --python-image option to override the
default, latest image with any other version released previously
until we diagnose and fix the real issue.

4 days agoFix timestamp defaults for sensorinstance (#24638)
Daniel Standish [Fri, 24 Jun 2022 16:31:58 +0000 (09:31 -0700)] 
Fix timestamp defaults for sensorinstance (#24638)

Constant values were used where callables were intended.

4 days agoHave consistent types between the ORM and the migration files (#24044)
Ephraim Anierobi [Fri, 24 Jun 2022 15:51:01 +0000 (16:51 +0100)] 
Have consistent types between the ORM and the migration files (#24044)

We currently don't compare column types between ORM and the migration files. Some columns in the migration files have different types from the same columns in the ORM.
Here, I made effort to match the types in migration files with the
 types in ORM, using the migration files as the source of truth in most cases.

I couldn't convert the MySQL VARCHAR collation in db(utf8_bin) to use the one in ORM(utf8mb3_bin). It seems it's not possible to convert a collation of an already existing column in MySQL.

4 days agoExternalTaskSensor respects soft_fail if the external task enters a failed_state...
Andrew Gibbs [Fri, 24 Jun 2022 13:50:13 +0000 (14:50 +0100)] 
ExternalTaskSensor respects soft_fail if the external task enters a failed_state (#23647)

* Respecting soft_fail in ExternalTaskSensor when the upstream tasks are in the failed state (#19754)

- Changed behaviour of sensor to as above to respect soft_fail
- Added tests of new soft_fail behaviour (#19754)
- Added newsfragment and improved sensor docstring

4 days agoAdd %z for %(asctime)s to fix timezone for logs on UI (#24373)
Ko HanJong [Fri, 24 Jun 2022 13:17:24 +0000 (22:17 +0900)] 
Add %z for %(asctime)s to fix timezone for logs on UI (#24373)

4 days agoAdd --platform to breeze shell and start-airflow (#24626)
Tzu-ping Chung [Fri, 24 Jun 2022 10:12:14 +0000 (18:12 +0800)] 
Add --platform to breeze shell and start-airflow (#24626)

This allows Breeze to run images of another platform, such as AMD64 on
an ARM machine.

4 days agoAdd gcp_conn_id argument to GoogleDriveToLocalOperator (#24622)
aspain [Fri, 24 Jun 2022 09:40:05 +0000 (05:40 -0400)] 
Add gcp_conn_id argument to GoogleDriveToLocalOperator (#24622)

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
4 days agoLimit azure-servicebus to not be used on ARM (#24635)
Jarek Potiuk [Fri, 24 Jun 2022 09:39:10 +0000 (11:39 +0200)] 
Limit azure-servicebus to not be used on ARM (#24635)

Azure service bus uses uamqp which does not build for ARM architecture
and we need to disable it as a dependency for ARM.

4 days agodocs: sqlalchemy link update (#24627)
Itay [Fri, 24 Jun 2022 08:12:22 +0000 (11:12 +0300)] 
docs: sqlalchemy link update (#24627)

following to this change: https://github.com/apache/airflow/pull/22114

4 days agoupdate `task-generated mapping` example (#23424)
aspain [Fri, 24 Jun 2022 04:28:54 +0000 (00:28 -0400)] 
update `task-generated mapping` example (#23424)

Co-authored-by: James Timmins <jameshtimmins@gmail.com>
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
5 days agoWe now need at least Flask-WTF 0.15 (#24621)
Ash Berlin-Taylor [Thu, 23 Jun 2022 19:58:07 +0000 (20:58 +0100)] 
We now need at least Flask-WTF 0.15 (#24621)

We upgraded flask and werkzeug in #24399, and updated the constraints,
but not everyone uses them (such as me in my local virtual environment
when developing) so the min version in setup.cfg has to match as well

5 days agoUpgrade FAB to 4.1.2 (#24619)
Jarek Potiuk [Thu, 23 Jun 2022 19:56:28 +0000 (21:56 +0200)] 
Upgrade FAB to 4.1.2 (#24619)

5 days agoLay the groundwork for migrating Airflow CLI to Rich+Click (#24590)
blag [Thu, 23 Jun 2022 16:28:17 +0000 (09:28 -0700)] 
Lay the groundwork for migrating Airflow CLI to Rich+Click (#24590)

This is the first PR of what will be a series of PRs breaking up #22613 into smaller, more reviewable chunks. The end result will be rewriting the existing `airflow` CLI to use Click instead of argparse. For motivation, please see #22708.

This PR installs Click, adds constraints to Rich_Click so we can rely on some nice features in recent versions of that, adds a new barebones `airflow-ng` console script, and tweaks some CLI internals to be more flexible between argparse and Click.

To see how this initial groundwork will be used by future PRs, see #22613, and to see how some of this will be used please see #24591.

5 days agoImplement Azure Service Bus Queue Operators (#24038)
Bharanidharan [Thu, 23 Jun 2022 15:52:47 +0000 (21:22 +0530)] 
Implement Azure Service Bus Queue Operators (#24038)

Implemented Azure Service Bus Queue based Operator's to create queue, send message to the queue and receive message(list of message or batch message) and delete queue in azure service
- Added `AzureServiceBusCreateQueueOperator`
- Added `AzureServiceBusSendMessageOperator`
- Added  `AzureServiceBusReceiveMessageOperator`
- Added `AzureServiceBusDeleteQueueOperator`
- Added Example DAG
- Added Documentation
- Added hooks and connection type in - provider yaml file
- Added unit Test case, doc strings

5 days agoUse sql_alchemy_conn for celery result backend when result_backend is not set (#24496)
Aakcht [Thu, 23 Jun 2022 15:34:21 +0000 (18:34 +0300)] 
Use sql_alchemy_conn for celery result backend when result_backend is not set (#24496)

5 days agoFix usage of `click.get_terminal_size()` (#24616)
Kaxil Naik [Thu, 23 Jun 2022 12:57:36 +0000 (13:57 +0100)] 
Fix usage of `click.get_terminal_size()` (#24616)

We were ignoring mypy error instead of fixing it.

click had removed `get_terminal_size` and recommend using `shutil.get_terminal_size`

5 days agoFix errors revealed on autoupgrade of breeze (#24612)
Jarek Potiuk [Thu, 23 Jun 2022 00:37:40 +0000 (02:37 +0200)] 
Fix errors revealed on autoupgrade of breeze (#24612)

Recent changes to Breeze cause it to fail in certain situations,
especially at self-upgrade (which was generated by today's
upgrade with rich-click).

There were two problems:

* docker volume inspect missed 'volume' and it caused sometimes
  failures in CI

* inputimeout dependency was missing after recent update to
  pre-commit venvs

5 days agoAdd note about image regeneration in June 2022 (#24524)
Jarek Potiuk [Thu, 23 Jun 2022 00:27:54 +0000 (02:27 +0200)] 
Add note about image regeneration in June 2022 (#24524)

The note is about image changes after refreshing are now better
organized (around date of the change) - this should be more useful
by the users who will look why their images have been refreshed.

Related to: #24516

5 days agoFix recording console for new rich-click 1.5 (#24611)
Jarek Potiuk [Wed, 22 Jun 2022 23:06:23 +0000 (01:06 +0200)] 
Fix recording console for new rich-click 1.5 (#24611)

6 days agoUpgrade FAB to 4.1.1 (#24399)
Jarek Potiuk [Wed, 22 Jun 2022 21:26:28 +0000 (23:26 +0200)] 
Upgrade FAB to 4.1.1 (#24399)

* Upgrade FAB to 4.1.1

The Flask Application Builder have been updated recently to
support a number of newer dependencies. This PR is the
attempt to migrate FAB to newer version.

This includes:

* update setup.py and setup.cfg upper and lower bounds to
  account for proper version of dependencies that
  FAB < 4.0.0 was blocking from upgrade
* added typed Flask application retrieval with a custom
  application fields available for MyPy typing checks.
* fix typing to account for typing hints added in multiple
  upgraded libraries optional values and content of request
  returned as Mapping
* switch to PyJWT 2.* by using non-deprecated "required" claim as
  list rather than separate fields
* add possibiliyt to install providers without constraints
  so that we could avoid errors on conflicting constraints when
  upgrade-to-newer-dependencies is used
* add pre-commit to check that 2.4+ only get_airflow_app is not
  used in providers
* avoid Bad Request in case the request sent to Flask 2.0 is not
  JSon content type
* switch imports of internal classes to direct packages
  where classes are available rather than from "airflow.models" to
  satisfy MyPY
* synchronize changes of FAB Security Manager 4.1.1 with our copy
  of the Security Manager.
* add error handling for a few "None" cases detected by MyPY
* corrected test cases that were broken by immutability of
  Flask 2 objects and better escaping done by Flask 2
* updated test cases to account for redirection to "path" rather
  than full URL by Flask2

Fixes: #22397

* fixup! Upgrade FAB to 4.1.1

6 days agoMigrate jsx files that affect run/task selection to tsx (#24509)
Brent Bovenzi [Wed, 22 Jun 2022 19:30:24 +0000 (15:30 -0400)] 
Migrate jsx files that affect run/task selection to tsx (#24509)

* convert all useSelection files to ts

Update grid data ts, remove some anys

* yarn, lint and tests

* convert statusbox to ts

* remove some anys, update instance tooltip

* fix types

* remove any, add comment for global vars

* fix url selection and grid/task defaults

* remove React.FC declarations

* specify tsconfig file path

* remove ts-loader

6 days agoUpdate breeze mypy check documentation (#24609)
Igor Tavares [Wed, 22 Jun 2022 19:28:27 +0000 (16:28 -0300)] 
Update breeze mypy check documentation (#24609)

6 days agoPatch sql_alchemy_conn if old Postgres schemes used (#24569)
Sergey Lyapustin [Wed, 22 Jun 2022 11:06:57 +0000 (13:06 +0200)] 
Patch sql_alchemy_conn if old Postgres schemes used (#24569)

6 days agoPattern parameter in S3ToSnowflakeOperator (#24571)
Andrey Anshin [Wed, 22 Jun 2022 07:49:02 +0000 (11:49 +0400)] 
Pattern parameter in S3ToSnowflakeOperator (#24571)

6 days agoRemove special serde logic for mapped op_kwargs (#23860)
Tzu-ping Chung [Wed, 22 Jun 2022 07:48:50 +0000 (15:48 +0800)] 
Remove special serde logic for mapped op_kwargs (#23860)

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
6 days agofix TestLocalTaskJob tests (#24432)
Andrey Anshin [Wed, 22 Jun 2022 06:15:49 +0000 (10:15 +0400)] 
fix TestLocalTaskJob tests (#24432)

6 days agoUpdate providers to use functools compat for ``cached_property`` (#24582)
Josh Fell [Wed, 22 Jun 2022 02:02:12 +0000 (22:02 -0400)] 
Update providers to use functools compat for ``cached_property`` (#24582)

6 days agoAmazon appflow (#24057)
Igor Tavares [Tue, 21 Jun 2022 23:14:46 +0000 (20:14 -0300)] 
Amazon appflow (#24057)

* Add Amazon AppFlow hook.

* Add Amazon AppFlow operators.

* Add Amazon AppFlow examples.

* Add Amazon Appflow docs.

* Apply comments/docs patterns.

* Removing the "private" attribute signal and more.

* Fix task_ids for example_appflow.

* Move datetime_to_epoch() to utils and more.

* Fix the AppflowBaseOperator name.

* Ignore AppflowBaseOperator during structure check.

* test_short_circuit refactor.

* Add get_airflow_version.

* Update airflow/providers/amazon/aws/hooks/appflow.py

Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
* Update airflow/providers/amazon/aws/operators/appflow.py

Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
* Update airflow/providers/amazon/aws/operators/appflow.py

Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
* Update airflow/providers/amazon/aws/operators/appflow.py

Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
* Update airflow/providers/amazon/aws/operators/appflow.py

Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
* Update airflow/providers/amazon/aws/operators/appflow.py

Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
* Addressing Josh's requests.

* Add cached_property to AppflowHook

* Update airflow/providers/amazon/aws/hooks/appflow.py

Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
* Update airflow/providers/amazon/aws/operators/appflow.py

Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
* Update airflow/providers/amazon/aws/operators/appflow.py

Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
* Update Josh's comment.

* Update cached_property import.

* Fix mypy.

Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
7 days agoAdd DagWarning model, and a check for missing pools (#23317)
Daniel Standish [Tue, 21 Jun 2022 18:55:20 +0000 (11:55 -0700)] 
Add DagWarning model, and a check for missing pools (#23317)

We can use this to track (and surface to the user in the web UI) configuration problems that don't rise to the level of failing the dag parse, such tasks that reference nonexistent pools.

7 days agoAdd test_connection method to Trino hook (#24583)
Phani Kumar [Tue, 21 Jun 2022 18:09:15 +0000 (23:39 +0530)] 
Add test_connection method to Trino hook (#24583)

7 days agoFix occasional test failures where pid matches part of another PID (#24584)
Jarek Potiuk [Tue, 21 Jun 2022 17:52:27 +0000 (19:52 +0200)] 
Fix occasional test failures where pid matches part of another PID (#24584)

Solves problems that intermittently fail our builds:

```
  '135' is contained here:
    ith pid: 10135
  ?            +++
```

7 days agoSwitch to building images in parallell (#24580)
Jarek Potiuk [Tue, 21 Jun 2022 15:58:54 +0000 (17:58 +0200)] 
Switch to building images in parallell (#24580)

In the new Breeze, switching to using parallelism is a ... breeze.

This PR adds the capability of building the images in parallel in Breeze
locally - for breeze command, but also uses this capability to build the
images in parallel in our CI. Our builds are always executed on
powerful, big machines with lots of CPU and docker run in memory
filesystem with 32GB RAM, so it should be possible to run all builds in
parallel on a single machine rather then spin off parallel machines to
run the builds using the matrix strategy of Github Actions.

Generally speaking - this will either speed up or get 4x cost saving for
the build steps for all the "full test needed" PRs as well as all the
main builds.

There are a number of savings and improvements we can achieve this way:

1) less overhead for starting and runnning the machines
2) seems that with the new buildkit, the parallel builds are not
   suffering from some sequential locks (as it used to be, so
   we are basically do the same job using 25% resources for building
   the images.
3) we will stop having random "one image failed to build" cases - they
   will all either fail or succeed.
4) Less checks in the output
5) Production builds will additionally gain from single CI image
   pulled in order to perform the preparation of the packages
   and single package preparation step - it will save 4-5 minutes
   per image.

The disadvantage is a less clear output of such parallel build where
outputs from multiple builds will be interleaved in one CI output.

7 days agoUse found pod for deletion in KubernetesPodOperator (#22092)
Michael Petro [Tue, 21 Jun 2022 15:51:20 +0000 (11:51 -0400)] 
Use found pod for deletion in KubernetesPodOperator (#22092)

Due to bad user configuration, it's possible that pod creation fails because pod with name already exists.  Then in cleanup, the pod that was already there is deleted.  When we use find_pod it looks up based on more than name, so we are confident if we found a pod there it's safe to delete.

7 days agoAdded instructions on what to do if your command images are regenerated (#24581)
Jarek Potiuk [Tue, 21 Jun 2022 13:59:43 +0000 (15:59 +0200)] 
Added instructions on what to do if your command images are regenerated (#24581)

In case there are conflicting changes to breeze command in several
PRs, you might get conflicting images printed. in such case you
should run `breeze regenerate-command-images`

7 days agoFix typo (#24568)
Jon Crall [Tue, 21 Jun 2022 13:13:33 +0000 (09:13 -0400)] 
Fix typo (#24568)

7 days agoSwitch Markdown engine to markdown-it-py (#19702)
Torbjørn Vatn [Tue, 21 Jun 2022 09:24:13 +0000 (11:24 +0200)] 
Switch Markdown engine to markdown-it-py (#19702)

7 days agoChange requests intersphinx inventory to 'stable' (#24575)
Tzu-ping Chung [Tue, 21 Jun 2022 07:40:23 +0000 (15:40 +0800)] 
Change requests intersphinx inventory to 'stable' (#24575)

It was using 'master', but the URL is no longer available. We could use
'main', but 'stable' seems much more sensible.

8 days agoAdd `on_kill()` to kill Trino query if the task is killed (#24559)
Phani Kumar [Mon, 20 Jun 2022 18:39:02 +0000 (00:09 +0530)] 
Add `on_kill()` to kill Trino query if the task is killed (#24559)

This PR is a follow up to PR #24415.
It adds on_kill method to the TrinoOperator to kill Trino query if Airflow task is killed

8 days agoClarify that users should not use Maria DB (#24556)
Jarek Potiuk [Mon, 20 Jun 2022 12:45:16 +0000 (14:45 +0200)] 
Clarify that users should not use Maria DB (#24556)

8 days agoFix the error caused by passing unused context in JiraSensor.poke (#23352)
Kian Eliasi [Mon, 20 Jun 2022 09:23:19 +0000 (13:53 +0430)] 
Fix the error caused by passing unused context in JiraSensor.poke (#23352)

* Fix the error caused by passing unused context in JiraSensor.poke

The sample ticket object used in the test is mocked more realistically and the original field_checker_func is used.

8 days agoMigrate Google example DAG oracle_to_gcs to new design AIP-47 (#24542)
Chenglong Yan [Mon, 20 Jun 2022 09:05:35 +0000 (17:05 +0800)] 
Migrate Google example DAG oracle_to_gcs to new design AIP-47 (#24542)

related: #22447, #22430

8 days agoUpdate description of installing providers separately from core (#24454)
Jarek Potiuk [Mon, 20 Jun 2022 08:09:56 +0000 (10:09 +0200)] 
Update description of installing providers separately from core (#24454)

The description of how to install providers separately from the
core was wrong. It suggested installing main constraints, but
in fact we are not able to give any guarantees when installing
providers this way so no constraints can guarantee consistent
set of dependencies and the user is on their own in this case.

This PR corrects the installation docs.

8 days agoDoc: Add hyperlinks to Github PRs for Release Notes (#24532)
Kaxil Naik [Mon, 20 Jun 2022 05:46:39 +0000 (06:46 +0100)] 
Doc: Add hyperlinks to Github PRs for Release Notes (#24532)

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
8 days agoAdd TrinoOperator (#24415)
Phani Kumar [Sun, 19 Jun 2022 23:08:14 +0000 (04:38 +0530)] 
Add TrinoOperator (#24415)

* Add TrinoOperator

* Enable multi query execution with TrinoOperator

8 days agoMigrate Google example DAG mysql_to_gcs to new design AIP-47 (#24540)
Chenglong Yan [Sun, 19 Jun 2022 22:56:08 +0000 (06:56 +0800)] 
Migrate Google example DAG mysql_to_gcs to new design AIP-47 (#24540)

related: #22447, #22430

8 days agoRe-serialize all DAGs on 'airflow db upgrade' (#24518)
Tzu-ping Chung [Sun, 19 Jun 2022 22:42:08 +0000 (06:42 +0800)] 
Re-serialize all DAGs on 'airflow db upgrade' (#24518)

8 days agoMake extra_args in S3Hook immutable between calls (#24527)
Andrey Anshin [Sun, 19 Jun 2022 22:41:36 +0000 (02:41 +0400)] 
Make extra_args in S3Hook immutable between calls (#24527)

8 days agoConvert sftp hook to use paramiko instead of pysftp (#24512)
Paul Williams [Sun, 19 Jun 2022 22:40:25 +0000 (18:40 -0400)] 
Convert sftp hook to use paramiko instead of pysftp (#24512)

8 days agoConvert SQS Sample DAG to System Test (#24513)
D. Ferruzzi [Sun, 19 Jun 2022 22:16:02 +0000 (15:16 -0700)] 
Convert SQS Sample DAG to System Test (#24513)

8 days agoMigrate Google search_ads DAG to new design AIP-47 (#24298)
Chenglong Yan [Sun, 19 Jun 2022 22:15:39 +0000 (06:15 +0800)] 
Migrate Google search_ads DAG to new design AIP-47 (#24298)

related: #22447, #22430

8 days agoClose the MySQL connections once operations are done. (#24508)
Pavel [Sun, 19 Jun 2022 22:15:01 +0000 (18:15 -0400)] 
Close the MySQL connections once operations are done. (#24508)

8 days agoGCSDeleteObjectsOperator empty prefix bug fix (#24353)
Daniel Lavedonio de Lima [Sun, 19 Jun 2022 22:07:56 +0000 (19:07 -0300)] 
GCSDeleteObjectsOperator empty prefix bug fix (#24353)

8 days agoMigrate Google gcs_to_sheets DAG to new design AIP-47 (#24501)
Chenglong Yan [Sun, 19 Jun 2022 22:06:46 +0000 (06:06 +0800)] 
Migrate Google gcs_to_sheets DAG to new design AIP-47 (#24501)

related: #22430, #22447

9 days agofix: RedshiftDataHook and RdsHook not use cached connection (#24387)
Andrey Anshin [Sun, 19 Jun 2022 21:32:51 +0000 (01:32 +0400)] 
fix: RedshiftDataHook and RdsHook not use cached connection (#24387)

9 days agoAdd missing types to FSHook (#24470)
Léni Marvaud [Sun, 19 Jun 2022 21:27:30 +0000 (23:27 +0200)] 
Add missing types to FSHook (#24470)

Fix mypy error :
```
error: Call to untyped function "FSHook" in typed context  [no-untyped-call]
```

When using :
```python
fs_hook = FSHook(fs_conn_id)
```

9 days agoFix bugs in URI constructor for MySQL connection (#24320)
Maksim [Sun, 19 Jun 2022 21:25:59 +0000 (00:25 +0300)] 
Fix bugs in URI constructor for MySQL connection (#24320)

* Fix bugs in URI constructor for MySQL connection

* Update unit tests

9 days agoAdd DeprecationWarning for column_transformations parameter in AutoML (#24467)
Maksim [Sun, 19 Jun 2022 21:25:04 +0000 (00:25 +0300)] 
Add DeprecationWarning for column_transformations parameter in AutoML (#24467)

9 days agoUpdate flask-appbuilder authlib/oauth dependency (#24516)
Jarek Potiuk [Sun, 19 Jun 2022 19:28:19 +0000 (21:28 +0200)] 
Update flask-appbuilder authlib/oauth dependency (#24516)

The dependency we have for flask-appbuilder oauth authentication
(for github/google authentication) should follow the limits
that flask-appbuilder current version has. We added authlib there
but apparently FAB currently limits authlib to <= 1.0 - we should
follow fab rather than have our own dependency here.

This has been pointed out in
https://github.com/dpgaspar/Flask-AppBuilder/issues/1861

9 days agoPreserve original order of providers' connection extra fields in UI (#24425)
Andrey Anshin [Sun, 19 Jun 2022 18:32:27 +0000 (22:32 +0400)] 
Preserve original order of providers' connection extra fields in UI (#24425)

* Preserve original order of providers' connection extra fields

* Sort widgets before printed in cli command `airflow providers widgets`