Improve caching for multi-platform images. (#23562)
authorJarek Potiuk <jarek@potiuk.com>
Mon, 9 May 2022 21:02:25 +0000 (23:02 +0200)
committerGitHub <noreply@github.com>
Mon, 9 May 2022 21:02:25 +0000 (23:02 +0200)
commit9a6baab5a271b28b6b3cbf96ffa151ac7dc79013
tree36b61209ac2bc24b04fe7c7604500216c0346ea5
parent46c1c006f747798f7c80f615d791553f25dd1770
Improve caching for multi-platform images. (#23562)

This is another attempt to improve caching performance for
multi-platform images as the previous ones were undermined by a
bug in buildx multiplatform cache-to implementattion that caused
the image cache to be overwritten between platforms,
when multiple images were build.

The bug is created for the buildx behaviour at
https://github.com/docker/buildx/issues/1044 and until it is fixed
we have to prpare separate caches for each platform and push them
to separate tags.

That adds a bit overhead on the building step, but for now it is
the simplest way we can workaround the bug if we do not want to
manually manipulate manifests and images.
46 files changed:
.github/workflows/build-images.yml
.github/workflows/ci.yml
BREEZE.rst
CONTRIBUTING.rst
Dockerfile
Dockerfile.ci
IMAGES.rst
INSTALL
dev/REFRESHING_CI_CACHE.md
dev/breeze/doc/adr/0005-preventing-using-contributed-code-when-building-images.md
dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
dev/breeze/src/airflow_breeze/commands/production_image_commands.py
dev/breeze/src/airflow_breeze/commands/release_management_commands.py
dev/breeze/src/airflow_breeze/global_constants.py
dev/breeze/src/airflow_breeze/params/_common_build_params.py [new file with mode: 0644]
dev/breeze/src/airflow_breeze/params/build_ci_params.py
dev/breeze/src/airflow_breeze/params/build_prod_params.py
dev/breeze/src/airflow_breeze/params/shell_params.py
dev/breeze/src/airflow_breeze/utils/common_options.py
dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
dev/breeze/src/airflow_breeze/utils/platforms.py [new file with mode: 0644]
dev/breeze/src/airflow_breeze/utils/pulll_image.py
dev/breeze/src/airflow_breeze/utils/registry.py
dev/breeze/src/airflow_breeze/utils/run_utils.py
dev/breeze/tests/test_docker_command_utils.py
dev/provider_packages/prepare_provider_packages.py
dev/refresh_images.sh
docs/apache-airflow/installation/installing-from-pypi.rst
images/breeze/output-build-image.svg
images/breeze/output-build-prod-image.svg
images/breeze/output-commands-hash.txt
images/breeze/output-commands.svg
images/breeze/output-generate-constraints.svg
images/breeze/output-shell.svg
images/breeze/output-start-airflow.svg
images/breeze/output-verify-provider-packages.svg
scripts/ci/docker-compose/_docker.env
scripts/ci/docker-compose/base.yml
scripts/ci/docker-compose/devcontainer.env
scripts/ci/libraries/_initialization.sh
scripts/docker/common.sh
scripts/docker/install_additional_dependencies.sh
scripts/docker/install_airflow.sh
scripts/docker/install_airflow_dependencies_from_branch_tip.sh
scripts/docker/install_from_docker_context_files.sh
scripts/in_container/run_generate_constraints.sh