airflow.git
4 months agoTests for provider code structure (#23351)
Bartłomiej Hirsz [Sun, 8 May 2022 19:32:26 +0000 (21:32 +0200)] 
Tests for provider code structure (#23351)

Improved test for code structure that can be re-used among various providders.

4 months agoAdd slim images to release process (#23391)
Jarek Potiuk [Sat, 7 May 2022 21:53:11 +0000 (23:53 +0200)] 
Add slim images to release process (#23391)

This PR adds slim images to release process of Airflow.

Those images are small as they do not contain any extras.

Fixes: #20849

4 months agoFix _PIP_ADDITIONAL_REQUIREMENTS case for docker-compose (#23517)
Jarek Potiuk [Sat, 7 May 2022 14:17:48 +0000 (16:17 +0200)] 
Fix _PIP_ADDITIONAL_REQUIREMENTS case for docker-compose (#23517)

Recent versions of Airflow do not allow to run `pip install` as
root but the `init` job runs as root so when the variable
_PIP_ADDITIONAL_REQUIREMENTS is set, the init container fails.

This PR forces _PIP_ADDITIONAL_REQUIREMENTS to be empty for the init
job.

4 months agoRefactor Breeze to group related methods and classes together (#23556)
Jarek Potiuk [Sat, 7 May 2022 13:56:34 +0000 (15:56 +0200)] 
Refactor Breeze to group related methods and classes together (#23556)

This change refactors Breeze classes to more consistent approach.

* The "commands" package only contains commands
* All Parameters (BuildCi, BuildProd, BuildDoc, Shell) are now
  in "params" package
* Required/Optional Build args are now members of the
  BuildCiParams, BuildProdParams which makes the params
  much more self-contained..
* All utils are in "utils" package

This helps with avoiding circular imports (all utios are now
standalone and do not use any of the commands.

Co-authored-by: eladkal <45845474+eladkal@users.noreply.github.com>
4 months agoAdd IPV6 form of the address in cassandra status check (#23537)
Jarek Potiuk [Sat, 7 May 2022 13:36:55 +0000 (15:36 +0200)] 
Add IPV6 form of the address in cassandra status check (#23537)

This PR fixes problem introduced in 3.0.26 of cassandra image which
adds square brackets around IP address regardless of its type.

The problem was workarounded by pinning cassandra to 3.0.25 in
the ##23522 as a quick fix, but this one introducec permanent,
future-proof solution.

Based on discussion in https://issues.apache.org/jira/browse/CASSANDRA-17612

Fixes: #23523

4 months agoAdd logging in to Github Registry for breeze pull (#23551)
Jarek Potiuk [Sat, 7 May 2022 10:24:05 +0000 (12:24 +0200)] 
Add logging in to Github Registry for breeze pull (#23551)

All of the Airlfow Images are Public in ghcr.io but default setting
for iamges is "private" and when users want to build CI workflows
in their forks, had to manually change their images to Public, so
that ci.yml workflow can pull the images prepared in the build-images
workflow.

This PR adds logging in for `breeze pull` command when GITHUB_TOKEN
is available, also the workflow gets packages: read permissions.

This way ci should works in forks of users without any action from
user except first-time workflow enabling.

4 months agoFix LocalFilesystemToS3Operator and S3CreateObjectOperator to support full s3://...
Vincent [Sat, 7 May 2022 09:19:45 +0000 (03:19 -0600)] 
Fix LocalFilesystemToS3Operator and S3CreateObjectOperator to support full s3:// style keys (#23180)

* Fix LocalFilesystemToS3Operator and S3CreateObjectOperator.
Support full s3:// style keys

* Fix spelling error

4 months agoChange chart annotation generator to use RELEASE_NOTES (#23549)
Jed Cunningham [Sat, 7 May 2022 09:15:25 +0000 (03:15 -0600)] 
Change chart annotation generator to use RELEASE_NOTES (#23549)

4 months agoUpdate the Athena Sample DAG and Docs (#23428)
D. Ferruzzi [Sat, 7 May 2022 06:28:44 +0000 (23:28 -0700)] 
Update the Athena Sample DAG and Docs (#23428)

* Update the Athena Sample DAG and Docs

4 months agoFix accidental including of providers in airflow package (#23552)
Jarek Potiuk [Sat, 7 May 2022 06:26:04 +0000 (08:26 +0200)] 
Fix accidental including of providers in airflow package (#23552)

The change #23454 accidentally remove INSTALL_PROVIDERS_FROM_SOURCES
setting to "false" which resulted in airflow package containing all
providers. This has been caught by our tests (but it was only
visible after merging)

This PR brings the variable back.

4 months agoReplace `pytest.mark.xfail` in Postgres tests (#23541)
eladkal [Fri, 6 May 2022 23:51:56 +0000 (02:51 +0300)] 
Replace `pytest.mark.xfail` in Postgres tests (#23541)

4 months agoSeperate provider verification as standalone breeze command (#23454)
Jarek Potiuk [Fri, 6 May 2022 22:47:00 +0000 (00:47 +0200)] 
Seperate provider verification as standalone breeze command (#23454)

This is another step in simplifying and converting to Python all of
the CI/local development tooling.

This PR separates out verification of providers as a separate
breeze command `verify-provider-packages`. It was previously part of
"prepare_provider_packages.py" but it has been now
extracted to a separate in-container python file and it was
wrapped with breeze's `verify-provider-packages` command.

No longer provider verification is run with "preparing provider docs"
nor "preparing provider packages" - it's a standaline command.

This command is also used in CI now to run the tests:

* all provider packages are built and created on CI together with
  airflow version
* the packages are installed inside the CI image and providers are
  verified
* the 2.1 version of Airflow is installed together with all 2.1
  - compatible providers and provider verification is run there too.

This all is much simpler now - we got rediof some 500 lines of bash
code again in favour of breeze python code.

Fixes: #23430

4 months agoTrinoHook add authentication via JWT token and Impersonation (#23116)
Pragya [Fri, 6 May 2022 19:45:33 +0000 (01:15 +0530)] 
TrinoHook add authentication via JWT token and Impersonation  (#23116)

* added trino authentication via JWT token and impersonation

* added test cases for jwt verification in trino

* added documenation for trino hook

4 months agoUpdate docs Amazon Glacier Docs (#23372)
Niko [Fri, 6 May 2022 18:03:24 +0000 (11:03 -0700)] 
Update docs Amazon Glacier Docs (#23372)

4 months agoChange approach to finding bad rows to LEFT OUTER JOIN. (#23528)
Ash Berlin-Taylor [Fri, 6 May 2022 16:02:27 +0000 (17:02 +0100)] 
Change approach to finding bad rows to LEFT OUTER JOIN. (#23528)

Rather than sub-selects (two for count, or one for the CREATE TABLE).

For a _large_ database (27m TaskInstances, 2m DagRuns) this takes the
time from 10minutes to around 3 minutes per table (we have 3) down to 3
minutes per table. (All times on Postgres.)

Before:

```sql
CREATE TABLE _airflow_moved__2_3__dangling__rendered_task_instance_fields AS
SELECT
  rendered_task_instance_fields.dag_id AS dag_id,
  rendered_task_instance_fields.task_id AS task_id,
  rendered_task_instance_fields.execution_date AS execution_date,
  rendered_task_instance_fields.rendered_fields AS rendered_fields,
  rendered_task_instance_fields.k8s_pod_yaml AS k8s_pod_yaml +
FROM
  rendered_task_instance_fields
WHERE
  NOT (
    EXISTS (
      SELECT
        1
      FROM
        task_instance
        JOIN dag_run ON dag_run.dag_id = task_instance.dag_id
        AND dag_run.run_id = task_instance.run_id
      WHERE
        rendered_task_instance_fields.dag_id = task_instance.dag_id
        AND rendered_task_instance_fields.task_id = task_instance.task_id
        AND rendered_task_instance_fields.execution_date = dag_run.execution_date
    )
  )
```

After:

```sql
CREATE TABLE _airflow_moved__2_3__dangling__rendered_task_instance_fields AS
SELECT
  rendered_task_instance_fields.dag_id AS dag_id,
  rendered_task_instance_fields.task_id AS task_id,
  rendered_task_instance_fields.execution_date AS execution_date,
  rendered_task_instance_fields.rendered_fields AS rendered_fields,
  rendered_task_instance_fields.k8s_pod_yaml AS k8s_pod_yaml +
FROM
  rendered_task_instance_fields
  LEFT OUTER JOIN dag_run ON rendered_task_instance_fields.dag_id = dag_run.dag_id
  AND rendered_task_instance_fields.execution_date = dag_run.execution_date
  LEFT OUTER JOIN task_instance ON dag_run.dag_id = task_instance.dag_id
  AND dag_run.run_id = task_instance.run_id
  AND rendered_task_instance_fields.task_id = task_instance.task_id
WHERE
  task_instance.dag_id IS NULL
  OR dag_run.dag_id IS NULL
;
```

4 months agoOnly count bad refs when `moved` table exists (#23491)
Daniel Standish [Fri, 6 May 2022 12:42:22 +0000 (05:42 -0700)] 
Only count bad refs when `moved` table exists (#23491)

This keeps the logic to fail without upgrading when (A) there are bad rows and
(B) the "moved" table already exists. But we optimize so that we don't count
the bad rows unless the "moved" table is there. Previously we counted always,
but the first time a user attempts upgrade, the tables won't be there so
there's no point in counting.

Instead what we do is skip right to the CTAS, creating the _airflow_moved
tables. If there aren't any rows in the "moved" table, then we delete the table
immediately.

Also included here is a delete optimization, where we join to the moved table
instead of running the not exists query again.

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Co-authored-by: Ash Berlin-Taylor <ash@apache.org>
4 months agoAdd `OpsgenieDeleteAlertOperator` (#23405)
eladkal [Fri, 6 May 2022 11:37:03 +0000 (14:37 +0300)] 
Add `OpsgenieDeleteAlertOperator` (#23405)

* Add `OpsgenieDeleteAlertOperator`

4 months agoFix cassandra to 3.0.25 (#23522)
Jarek Potiuk [Fri, 6 May 2022 10:25:45 +0000 (12:25 +0200)] 
Fix cassandra to 3.0.25 (#23522)

fix cassandra to 3.0.25 as latest 3.0 (3.0.26) does not start cleanly

4 months agoMove tests command in new breeze (#23445)
Joppe Vos [Fri, 6 May 2022 09:03:05 +0000 (11:03 +0200)] 
Move tests command in new breeze (#23445)

4 months agoExpand/collapse all groups (#23487)
Brent Bovenzi [Thu, 5 May 2022 18:20:22 +0000 (14:20 -0400)] 
Expand/collapse all groups (#23487)

* Add expand/collapse all groups button to Grid

* add tests

* add comments

* Switch to 2 icon buttons

Disable buttons if all groups are expanded or collapsed

* Update localStorage key

4 months agoReplace DummyOperator references in docs (#23502)
Leah E. Cole [Thu, 5 May 2022 15:26:14 +0000 (11:26 -0400)] 
Replace DummyOperator references in docs (#23502)

4 months agoChanged word 'the' instead 'his' (#23493)
Edith Puclla [Thu, 5 May 2022 15:06:35 +0000 (10:06 -0500)] 
Changed word 'the' instead 'his' (#23493)

4 months agoUse kubernetes queue in kubernetes hybrid executors (#23048)
Tanel Kiis [Thu, 5 May 2022 10:23:18 +0000 (13:23 +0300)] 
Use kubernetes queue in kubernetes hybrid executors (#23048)

When using "hybrid" executors (`CeleryKubernetesExecutor` or `LocalKubernetesExecutor`),
then the `clear_not_launched_queued_tasks` mechnism in the `KubernetesExecutor` can
reset the queued tasks, that were given to the other executor.

`KuberneterExecutor` should limit itself to the configured queue when working in the
"hybrid" mode.

4 months agoAdd Stackdriver assets and migrate system tests to AIP-47 (#23320)
Bartłomiej Hirsz [Wed, 4 May 2022 21:49:58 +0000 (23:49 +0200)] 
Add Stackdriver assets and migrate system tests to AIP-47 (#23320)

Change-Id: I6f751e6576f57a89a5145aeb05f506da8a22b379

Co-authored-by: Bartlomiej Hirsz <bartomiejh@google.com>
4 months agoAdd doc and example dag for Amazon SQS Operators (#23312)
Niko [Wed, 4 May 2022 21:45:38 +0000 (14:45 -0700)] 
Add doc and example dag for Amazon SQS Operators (#23312)

4 months agoAdds resultBackendSecretName warning in Helm production docs (#23307)
Rafael Passos [Wed, 4 May 2022 21:28:15 +0000 (18:28 -0300)] 
Adds resultBackendSecretName warning in Helm production docs (#23307)

4 months agoCloudTasks assets & system tests migration (AIP-47) (#23282)
Bartłomiej Hirsz [Wed, 4 May 2022 20:42:45 +0000 (22:42 +0200)] 
CloudTasks assets & system tests migration (AIP-47) (#23282)

4 months agoAdd support for timezone as string in cron interval timetable (#23279)
Malthe Borch [Wed, 4 May 2022 20:41:34 +0000 (20:41 +0000)] 
Add support for timezone as string in cron interval timetable (#23279)

4 months agoTextToSpeech assets & system tests migration (AIP-47) (#23247)
Bartłomiej Hirsz [Wed, 4 May 2022 20:40:00 +0000 (22:40 +0200)] 
TextToSpeech assets & system tests migration (AIP-47) (#23247)

4 months agoFix literal cross product expansion (#23434)
Jed Cunningham [Wed, 4 May 2022 19:02:09 +0000 (13:02 -0600)] 
Fix literal cross product expansion (#23434)

4 months agoVisually distinguish task group summarys (#23488)
Brent Bovenzi [Wed, 4 May 2022 19:01:51 +0000 (15:01 -0400)] 
Visually distinguish task group summarys (#23488)

Bold task groups names and darken their bottom row border.

4 months agoEnsure the messages from migration job show up early (#23479)
Ash Berlin-Taylor [Wed, 4 May 2022 15:52:42 +0000 (16:52 +0100)] 
Ensure the messages from migration job show up early (#23479)

The default for python is to buffer stdout, which means that log lines
might now show up in the output straight away (until a certain number of
lines or number of bytes of output have been written) -- this is
especially problematic if the pre-migration checks taking a long time as
it makes it look like it has hung

4 months agoRemove color change for highly nested groups (#23482)
Brent Bovenzi [Wed, 4 May 2022 15:30:58 +0000 (11:30 -0400)] 
Remove color change for highly nested groups (#23482)

4 months agoRemove remaining Python3.6 references (#23474)
Ephraim Anierobi [Wed, 4 May 2022 14:23:00 +0000 (15:23 +0100)] 
Remove remaining Python3.6 references (#23474)

4 months agoAdd Python 3.10 trove classifier (#23464)
Jed Cunningham [Wed, 4 May 2022 00:12:47 +0000 (18:12 -0600)] 
Add Python 3.10 trove classifier (#23464)

4 months agoBump pre-commit hook versions (#22887)
Kamil Breguła [Tue, 3 May 2022 22:37:30 +0000 (00:37 +0200)] 
Bump pre-commit hook versions (#22887)

4 months agoMove non-opencontainer labeling of the image to breeze from Dockerfile (#23379)
Jarek Potiuk [Tue, 3 May 2022 22:15:52 +0000 (00:15 +0200)] 
Move non-opencontainer labeling of the image to breeze from Dockerfile (#23379)

* Extract "extra" labeling of the image to breeze from Dockerfile

Fixes: #21046

* Add more ArtifictHub-specific labels

Co-authored-by: Kamil Breguła <kamilbregula@apache.org>
5 months agoShow warning if '/' is used in a DAG run ID (#23106)
Tzu-ping Chung [Tue, 3 May 2022 21:22:12 +0000 (15:22 -0600)] 
Show warning if '/' is used in a DAG run ID (#23106)

5 months agoUnify approach for user questions asked in Breeze (#23335)
Jarek Potiuk [Tue, 3 May 2022 20:22:42 +0000 (22:22 +0200)] 
Unify approach for user questions asked in Breeze (#23335)

This change documents and unifies the approach we've taken for
the user inut handling when it comes to confirmation questions.

5 months agoDocs: Python 3.10 is now supported (#23457)
Jed Cunningham [Tue, 3 May 2022 19:00:50 +0000 (13:00 -0600)] 
Docs: Python 3.10 is now supported (#23457)

5 months agoOptimize 2.3.0 pre-upgrade check queries (#23458)
Daniel Standish [Tue, 3 May 2022 18:50:23 +0000 (11:50 -0700)] 
Optimize 2.3.0 pre-upgrade check queries (#23458)

We have to check for rows that are missing either corresponding TI or DR and move them out of table before adding FKs.  We were doing correlation in the JOIN condition but it appears postgres does *not* like this so here we move correlation to WHERE.

5 months agoFix `check_files.py` to work on new minor releases (#23287)
Jed Cunningham [Tue, 3 May 2022 18:33:05 +0000 (12:33 -0600)] 
Fix `check_files.py` to work on new minor releases (#23287)

5 months agoSupport annotations on volumeClaimTemplates (#23433)
Jed Cunningham [Tue, 3 May 2022 17:17:33 +0000 (11:17 -0600)] 
Support annotations on volumeClaimTemplates (#23433)

5 months agoAdd support for topologySpreadConstraints to Helm Chart (#22712)
Junyoung Park [Tue, 3 May 2022 17:15:36 +0000 (02:15 +0900)] 
Add support for topologySpreadConstraints to Helm Chart (#22712)

5 months agoFix code-snippets in google provider (#23438)
eladkal [Tue, 3 May 2022 16:10:00 +0000 (19:10 +0300)] 
Fix code-snippets in google provider (#23438)

5 months ago[FIX] remove python 3.6 (#23409)
raphaelauv [Tue, 3 May 2022 13:52:36 +0000 (15:52 +0200)] 
[FIX] remove python 3.6 (#23409)

5 months agoImprove verbose output of Breeze (#23446)
Jarek Potiuk [Tue, 3 May 2022 13:03:01 +0000 (15:03 +0200)] 
Improve verbose output of Breeze (#23446)

When you add --verbose or --dry-run options to breeze it will
print the commands it is executing (or is supposed to in dry-run
mode). The output contains environment variables as they
often contain crucial information to execute the command (for
example in docker-compose run commands it contains COMPOSE_FILE
variable which is the list of compose files that are used). This
is done in a fashion that you can copy the whole command and
execute it, but it very unfriendly for visual inspection as
all the variables were printed in one line and in semi-random
order and also the variables contained often all system variables
set by the shell before.

This change keeps the proerty of "we can copy&paste the command
and run it" but it improves the visual aspect of it:

1) each env variable is kept in one line
2) first all system variables are printed and then variables that
   were specifically added for this command
3) variables in each group are sorted alphabetically which helps
   in finding the variable you are looking for when you visually
   inspect the output.

5 months agoAdd backward compatibility for core__sql_alchemy_conn__cmd (#23441)
c-thiel [Tue, 3 May 2022 12:48:30 +0000 (14:48 +0200)] 
Add backward compatibility for core__sql_alchemy_conn__cmd (#23441)

5 months agoAdd fields to dagrun endpoint (#23440)
Karthikeyan Singaravelan [Tue, 3 May 2022 12:27:14 +0000 (17:57 +0530)] 
Add fields to dagrun endpoint (#23440)

* Add below fields to dagrun endpoint :

* data_interval_start
* data_interval_end
* last_scheduling_decision
* run_type

* Refactor hardcoded dates with constants.

5 months agoRefactor code references from tree to grid (#23254)
Brent Bovenzi [Tue, 3 May 2022 01:19:22 +0000 (21:19 -0400)] 
Refactor code references from tree to grid (#23254)

5 months agoImprove react www tests (#23329)
Brent Bovenzi [Mon, 2 May 2022 23:38:57 +0000 (19:38 -0400)] 
Improve react www tests (#23329)

* Add shared test wrapper & treeData placeholder

* remove console log

* move testUtils to /utils

* change unnamed export

5 months agoMark image as refreshed when pulled on CI (#23410)
Jarek Potiuk [Mon, 2 May 2022 16:13:44 +0000 (18:13 +0200)] 
Mark image as refreshed when pulled on CI (#23410)

One of the recent changes in Breeze (#23395) caused unnecessary
rebuilding of image when "build-docs" is run. On CI we build
image once and reuse it. However in case of Buld docs we missed
information that the image is "fresh" and we started rebuilding
it. This change marks the image as "refreshed" when it is pulled
with `--tag-as-latest` flag (which happens in CI).

5 months agoFix "breeze-legacy" after building images was removed (#23404)
Jarek Potiuk [Mon, 2 May 2022 14:26:05 +0000 (16:26 +0200)] 
Fix "breeze-legacy" after building images was removed (#23404)

The `breeze-legacy` stopped working after building images were
removed as few parameters were still checked for allowed values
but they were missing,

This PR fixes it by removing the parameters.

5 months agoCleanup Google provider CHANGELOG.rst (#23390)
eladkal [Mon, 2 May 2022 11:03:20 +0000 (14:03 +0300)] 
Cleanup Google provider CHANGELOG.rst (#23390)

5 months agoValidate DAG owner to be a string (#23359)
Karthikeyan Singaravelan [Mon, 2 May 2022 10:58:53 +0000 (16:28 +0530)] 
Validate DAG owner to be a string (#23359)

non-string values raise `AttributeError` as `task.owner.lower` is called with `task.owner` not being a string and the error is not passed as import error failing silently. Raise explicit error will be helpful to the user.

closes: #23343
related: #23343

5 months agoImprove handling of entry and exit to common Breeze commands (#23395)
Jarek Potiuk [Mon, 2 May 2022 10:56:46 +0000 (12:56 +0200)] 
Improve handling of entry and exit to common Breeze commands (#23395)

This PR improves handling of both entry and exit to common
Breeze commands:

* at entry all common commands check if rebuild of image is needed
* when you exit and there is an error from shell commands, rather
  than printing stack trace an error message is printed

5 months agoIgnore some files/directory when releasing source code (#23325)
Ephraim Anierobi [Mon, 2 May 2022 09:58:42 +0000 (10:58 +0100)] 
Ignore some files/directory when releasing source code (#23325)

5 months agoRemove tagging of `constraint-x-y` branch from release process (#23399)
Ephraim Anierobi [Mon, 2 May 2022 09:56:05 +0000 (10:56 +0100)] 
Remove tagging of `constraint-x-y` branch from release process (#23399)

5 months agoAdd better description for Breeze customization (#23397)
Jarek Potiuk [Mon, 2 May 2022 09:09:59 +0000 (11:09 +0200)] 
Add better description for Breeze customization (#23397)

5 months agoClarify 2.3.0 kubernetes min version is about library not cluster (#23398)
Jarek Potiuk [Mon, 2 May 2022 09:01:51 +0000 (11:01 +0200)] 
Clarify 2.3.0 kubernetes min version is about library not cluster (#23398)

It was not clear from the release notes that the minimum version
of the kubernetes was about the library rather than cluster version.

Also README was not updated with min versions of Kubernetes cluster
for 2.3.0 version.

5 months agoAllow multiline text in private key field for Snowflake (#23066)
Kamil Breguła [Sun, 1 May 2022 11:11:13 +0000 (13:11 +0200)] 
Allow multiline text in private key field for Snowflake (#23066)

5 months agoFix `version_added` for `[sensors] default_timeout` (#23388)
Jed Cunningham [Sun, 1 May 2022 11:08:21 +0000 (05:08 -0600)] 
Fix `version_added` for `[sensors] default_timeout` (#23388)

5 months agoadd auto refresh to dags home page (#22900)
Tal Nagar [Sun, 1 May 2022 10:59:30 +0000 (13:59 +0300)] 
add auto refresh to dags home page (#22900)

* add auto refresh to dags home page

* fix lint errors

* fix lint

* stop refresh when page is not focused or no active dag runs. change css for layout

* remove margin for refresh switch

* Update airflow/www/static/css/main.css

Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>
* Update airflow/www/static/js/dags.js

Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>
* Update airflow/www/static/js/dags.js

Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>
* fix text case

* Update airflow/www/static/js/dags.js

Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>
* add comment on refresh interval

* refactor last dag run handler date update

Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>
5 months agoFix attempting to reattach in `ECSOperator` (#23370)
Firass [Sun, 1 May 2022 10:58:13 +0000 (05:58 -0500)] 
Fix attempting to reattach in `ECSOperator` (#23370)

* Updated ecs_task_id on reattaching

5 months ago`DataprocHook`: Remove deprecated function `submit` (#23389)
eladkal [Sun, 1 May 2022 10:20:51 +0000 (13:20 +0300)] 
`DataprocHook`: Remove deprecated function `submit` (#23389)

5 months agoAdd missing steps to release process (#23384)
Jed Cunningham [Sun, 1 May 2022 10:09:53 +0000 (04:09 -0600)] 
Add missing steps to release process (#23384)

5 months agoChart: Update default airflow version to `2.3.0` (#23386)
Ephraim Anierobi [Sun, 1 May 2022 03:50:05 +0000 (04:50 +0100)] 
Chart: Update default airflow version to `2.3.0` (#23386)

5 months ago`2.3.0` has been released (#23385)
Ephraim Anierobi [Sun, 1 May 2022 03:41:07 +0000 (04:41 +0100)] 
`2.3.0` has been released (#23385)

5 months agoUpdate missing `version_added` in config.yml (#23387)
Ephraim Anierobi [Sun, 1 May 2022 03:37:11 +0000 (04:37 +0100)] 
Update missing `version_added` in config.yml (#23387)

5 months agoClarify `reattach_on_restart` behavior (#23377)
Jed Cunningham [Sat, 30 Apr 2022 23:31:27 +0000 (17:31 -0600)] 
Clarify `reattach_on_restart` behavior (#23377)

5 months agoFix deriving of PyPI branch from airflow version (#23380)
Jarek Potiuk [Sat, 30 Apr 2022 23:24:53 +0000 (01:24 +0200)] 
Fix deriving of PyPI branch from airflow version (#23380)

The regexp expression for deriving right branch from version was
missing . for version numbers.

5 months agoUpdate multiplatform doc image tagging to use regctl (#23383)
Jarek Potiuk [Sat, 30 Apr 2022 23:24:20 +0000 (01:24 +0200)] 
Update multiplatform doc image tagging to use regctl (#23383)

You cannot use 'docker tag' to move multiplatform image, instead
we should use regctl.

5 months agoFew fixes in the providers release doc (#23382)
Jed Cunningham [Sat, 30 Apr 2022 22:33:15 +0000 (16:33 -0600)] 
Few fixes in the providers release doc (#23382)

5 months agoOverride pool for TaskInstance when pool is passed from cli. (#23258)
Karthikeyan Singaravelan [Sat, 30 Apr 2022 19:11:07 +0000 (00:41 +0530)] 
Override pool for TaskInstance when pool is passed from cli. (#23258)

5 months agoBigquery assets (#23165)
Wojciech Januszek [Sat, 30 Apr 2022 18:34:39 +0000 (20:34 +0200)] 
Bigquery assets (#23165)

5 months agoAdd YANKED to yanked releases of the cncf.kubernetes (#23378)
Jarek Potiuk [Sat, 30 Apr 2022 18:33:22 +0000 (20:33 +0200)] 
Add YANKED to yanked releases of the cncf.kubernetes (#23378)

5 months agoFix duplicated Kubernetes DeprecationWarnings (#23302)
Jed Cunningham [Sat, 30 Apr 2022 17:47:32 +0000 (11:47 -0600)] 
Fix duplicated Kubernetes DeprecationWarnings (#23302)

5 months agofix cli `airflow dags show` for mapped operator (#23339)
Ryan Hatter [Sat, 30 Apr 2022 17:46:08 +0000 (13:46 -0400)] 
fix cli `airflow dags show` for mapped operator (#23339)

5 months agoAdd doc notes for keyword-only args for `expand()` and `partial()` (#23373)
Josh Fell [Sat, 30 Apr 2022 17:37:31 +0000 (13:37 -0400)] 
Add doc notes for keyword-only args for `expand()` and `partial()` (#23373)

5 months agoAdd missing --for-production parameter for new breeze docs building (#23376)
Jarek Potiuk [Sat, 30 Apr 2022 11:55:11 +0000 (13:55 +0200)] 
Add missing --for-production parameter for new breeze docs building (#23376)

5 months agoPrepare documentation for cncf.kubernetes 4.0.1 release (#23374) providers-cncf-kubernetes/4.0.1 providers-cncf-kubernetes/4.0.1rc1
Jarek Potiuk [Sat, 30 Apr 2022 10:06:08 +0000 (12:06 +0200)] 
Prepare documentation for cncf.kubernetes 4.0.1 release (#23374)

5 months agoWhen exec fails in breeze we do not print stack-trace (#23342)
Jarek Potiuk [Sat, 30 Apr 2022 09:13:34 +0000 (11:13 +0200)] 
When exec fails in breeze we do not print stack-trace (#23342)

When you run exec and breeze is not running, there was a stack
trace printed rather than straightforward error message.

This fixes it - stacktrace is only printed now when verbose is
used. If not just error message is printed.

5 months agoAllow back script_location in Glue to be None (#23357)
Jarek Potiuk [Sat, 30 Apr 2022 09:12:58 +0000 (11:12 +0200)] 
Allow back script_location in Glue to be None (#23357)

5 months agoFix ``KubernetesPodOperator`` with `KubernetesExecutor`` on 2.3.0 (#23371)
Jed Cunningham [Fri, 29 Apr 2022 21:35:44 +0000 (15:35 -0600)] 
Fix ``KubernetesPodOperator`` with `KubernetesExecutor`` on 2.3.0 (#23371)

KubernetesPodOperator was mistakenly trying to reattach to it's
KubernetesExecutor worker, where it would get stuck watching itself for
logs. We will properly filter for KPO's only, and ignore
KubernetesExecutor workers for good measure.

5 months agoFix mssql in the new Breeze (#23368)
Jarek Potiuk [Fri, 29 Apr 2022 17:45:37 +0000 (19:45 +0200)] 
Fix mssql in the new Breeze (#23368)

The new Breeze did not use conditionally debian version to run
MsSQL docker compose.

This PR fixes it

5 months agoFix broken task instance link in xcom list (#23367)
Brent Bovenzi [Fri, 29 Apr 2022 17:43:32 +0000 (13:43 -0400)] 
Fix broken task instance link in xcom list (#23367)

* Fix broken task instance link in xcom list

Add execution date back to the xcom list to be able to pass to the `task_instance_link()` function.
Long term, we should swap out the execution_date param for run_id

* Make execution date a search column

5 months agoAllow for LOGGING_LEVEL=DEBUG (#23360)
GApfel [Fri, 29 Apr 2022 17:24:39 +0000 (19:24 +0200)] 
Allow for LOGGING_LEVEL=DEBUG (#23360)

5 months agoOrganize Tableau classes (#23353)
eladkal [Fri, 29 Apr 2022 16:03:23 +0000 (19:03 +0300)] 
Organize Tableau classes (#23353)

* Organize Tableau classes

5 months agoRemove redundant docstring in `BigQueryUpdateTableSchemaOperator` (#23349)
eladkal [Fri, 29 Apr 2022 16:02:42 +0000 (19:02 +0300)] 
Remove redundant docstring in `BigQueryUpdateTableSchemaOperator` (#23349)

5 months ago`S3ToGCSOperator`: Remove `dest_gcs_conn_id` (#23348)
eladkal [Fri, 29 Apr 2022 15:46:51 +0000 (18:46 +0300)] 
`S3ToGCSOperator`: Remove `dest_gcs_conn_id` (#23348)

5 months agoDataproc: Remove default value of `region` (#23350)
eladkal [Fri, 29 Apr 2022 15:46:21 +0000 (18:46 +0300)] 
Dataproc: Remove default value of `region` (#23350)

* `region` parameter has no default value.
  affected functions/classes:
  `DataprocHook.cancel_job`
  `DataprocCreateClusterOperator`
  `DataprocJobBaseOperator`

  * `DataprocJobBaseOperator`: order of parameters has changed

5 months agoFix regeneration of breeze screenshots (#23344)
Jarek Potiuk [Fri, 29 Apr 2022 15:43:22 +0000 (17:43 +0200)] 
Fix regeneration of breeze screenshots (#23344)

5 months agoStore grid view selection in url params (#23290)
Brent Bovenzi [Fri, 29 Apr 2022 13:51:18 +0000 (09:51 -0400)] 
Store grid view selection in url params (#23290)

* Add url params for dag_run_id and task_id

* Persist other search params

* simplify useSelection

* delete extra params

* remove API change

5 months agoFix connection test button (#23345)
Jed Cunningham [Fri, 29 Apr 2022 10:29:02 +0000 (04:29 -0600)] 
Fix connection test button (#23345)

The connection test button was always disabled if any of your hooks had
import errors, for example because of a missing module. This handles
that scenario.

5 months agoGoogle provider: Remove `bigquery_conn_id`, `google_cloud_storage_conn_id` (#23326)
eladkal [Fri, 29 Apr 2022 06:08:13 +0000 (09:08 +0300)] 
Google provider: Remove `bigquery_conn_id`, `google_cloud_storage_conn_id` (#23326)

* `bigquery_conn_id` is removed. Please use `gcp_conn_id`.
  affected classes:
  `BigQueryCheckOperator`
  `BigQueryCreateEmptyDatasetOperator`
  `BigQueryDeleteDatasetOperator`
  `BigQueryDeleteTableOperator`
  `BigQueryExecuteQueryOperator`
  `BigQueryGetDataOperator`
  `BigQueryHook`
  `BigQueryIntervalCheckOperator`
  `BigQueryTableExistenceSensor`
  `BigQueryTablePartitionExistenceSensor`
  `BigQueryToBigQueryOperator`
  `BigQueryToGCSOperator`
  `BigQueryUpdateTableSchemaOperator`
  `BigQueryUpsertTableOperator`
  `BigQueryValueCheckOperator`
  `GCSToBigQueryOperator`

* `google_cloud_storage_conn_id` is removed. Please use `gcp_conn_id`.
  affected classes:
  `ADLSToGCSOperator`
  `BaseSQLToGCSOperator`
  `CassandraToGCSOperator`
  `GCSBucketCreateAclEntryOperator`
  `GCSCreateBucketOperator`
  `GCSDeleteObjectsOperator`
  `GCSHook`
  `GCSListObjectsOperator`
  `GCSObjectCreateAclEntryOperator`
  `GCSToBigQueryOperator`
  `GCSToGCSOperator`
  `GCSToLocalFilesystemOperator`
  `LocalFilesystemToGCSOperator`

5 months agoCleaner default output when breeze starts (#23341)
Jarek Potiuk [Thu, 28 Apr 2022 22:06:51 +0000 (00:06 +0200)] 
Cleaner default output when breeze starts (#23341)

There was a bit of noise printed when Breeze started:

* information about branch/python/image/backend used
* information about actions performed (like fixing permissions)
* information that docke image build is not needed
* warnings about missing variables

This PR marks all the messages as "info" and only prints them
when --verbose flag is used and it adds default values for the
variables that generated warnings.

5 months agoHide some task instance attributes (#23338)
Brent Bovenzi [Thu, 28 Apr 2022 20:39:38 +0000 (16:39 -0400)] 
Hide some task instance attributes (#23338)

5 months agoresolving conflict (#23052)
Bowrna [Thu, 28 Apr 2022 19:31:49 +0000 (01:01 +0530)] 
resolving conflict (#23052)

5 months agoRemove confusion about upgrade-to-newer-dependencies breeze param (#23334)
Jarek Potiuk [Thu, 28 Apr 2022 19:12:35 +0000 (21:12 +0200)] 
Remove confusion about upgrade-to-newer-dependencies breeze param (#23334)

The "upgrade-to-newer-dependencies" in the image can take "false"
"true" and <RANDOM> value (the RANDOM value is used to always
invalidate docker cache).

This has been carried to Breeze command line, but this was a source
of major confusion as the name of the parameter suggest bool value.

The change modifies the parameter to be flag, and when the flag
is set the parameter is set to random during image building.

That should help with recent bug when image was always rebuilt
without checking if it should be rebuilt.

5 months agoFix update user auth stats (#23314)
blag [Thu, 28 Apr 2022 18:59:24 +0000 (11:59 -0700)] 
Fix update user auth stats (#23314)