Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Last update: Dec 28, 2022

Overview

Apache Airflow

Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows.

When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.

Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

Table of contents

Project Focus
Principles
Requirements
Getting started
Installing from PyPI
Official source code
Convenience packages
User Interface
Semantic versioning
Version Life Cycle
Support for Python and Kubernetes versions
Contributing
Who uses Apache Airflow?
Who Maintains Apache Airflow?
Can I use the Apache Airflow logo in my presentation?
Airflow merchandise
Links
Sponsors

Project Focus

Airflow works best with workflows that are mostly static and slowly changing. When the DAG structure is similar from one run to the next, it clarifies the unit of work and continuity. Other similar projects include Luigi, Oozie and Azkaban.

Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent (i.e., results of the task will be the same, and will not create duplicated data in a destination system), and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's Xcom feature). For high-volume, data-intensive tasks, a best practice is to delegate to external services specializing in that type of work.

Airflow is not a streaming solution, but it is often used to process real-time data, pulling data off streams in batches.

Principles

Dynamic: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically.
Extensible: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment.
Elegant: Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful Jinja templating engine.
Scalable: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers.

Requirements

Apache Airflow is tested with:

	Main version (dev)	Stable version (2.2.2)
Python	3.6, 3.7, 3.8, 3.9	3.6, 3.7, 3.8, 3.9
Kubernetes	1.20, 1.21	1.18, 1.19, 1.20
PostgreSQL	10, 11, 12, 13	9.6, 10, 11, 12, 13
MySQL	5.7, 8	5.7, 8
SQLite	3.15.0+	3.15.0+
MSSQL(Experimental)	2017, 2019

Note: MySQL 5.x versions are unable to or have limitations with running multiple schedulers -- please see the Scheduler docs. MariaDB is not tested/recommended.

Note: SQLite is used in Airflow tests. Do not use it in production. We recommend using the latest stable version of SQLite for local development.

Note: Python v3.10 is not supported yet. For details, see #19059.

Note: Airflow currently can be run on POSIX-compliant Operating Systems. For development it is regularly tested on fairly modern Linux Distros and recent versions of MacOS. On Windows you can run it via WSL2 (Windows Subsystem for Linux 2) or via Linux Containers. The work to add Windows support is tracked via #10388 but it is not a high priority. You should only use Linux-based distros as "Production" execution environment as this is the only environment that is supported. The only distro that is used in our CI tests and that is used in the Community managed DockerHub image is Debian Buster.

Getting started

Visit the official Airflow website documentation (latest stable release) for help with installing Airflow, getting started, or walking through a more complete tutorial.

Note: If you're looking for documentation for the main branch (latest development branch): you can find it on s.apache.org/airflow-docs.

For more information on Airflow Improvement Proposals (AIPs), visit the Airflow Wiki.

Documentation for dependent projects like provider packages, Docker image, Helm Chart, you'll find it in the documentation index.

Installing from PyPI

We publish Apache Airflow as apache-airflow package in PyPI. Installing it however might be sometimes tricky because Airflow is a bit of both a library and application. Libraries usually keep their dependencies open, and applications usually pin them, but we should do neither and both simultaneously. We decided to keep our dependencies as open as possible (in setup.py) so users can install different versions of libraries if needed. This means that pip install apache-airflow will not work from time to time or will produce unusable Airflow installation.

To have repeatable installation, however, we keep a set of "known-to-be-working" constraint files in the orphan constraints-main and constraints-2-0 branches. We keep those "known-to-be-working" constraints files separately per major/minor Python version. You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify correct Airflow tag/version/branch and Python versions in the URL.

Installing just Airflow:

Note: Only pip installation is currently officially supported.

While it is possible to install Airflow with tools like Poetry or pip-tools, they do not share the same workflow as pip - especially when it comes to constraint vs. requirements management. Installing via Poetry or pip-tools is not currently supported.

If you wish to install Airflow using those tools, you should use the constraint files and convert them to the appropriate format and workflow that your tool requires.

pip install 'apache-airflow==2.2.2' \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.2/constraints-3.7.txt"

Installing with extras (i.e., postgres, google)

pip install 'apache-airflow[postgres,google]==2.2.2' \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.2/constraints-3.7.txt"

For information on installing provider packages, check providers.

Official source code

Apache Airflow is an Apache Software Foundation (ASF) project, and our official source code releases:

Follow the ASF Release Policy
Can be downloaded from the ASF Distribution Directory
Are cryptographically signed by the release manager
Are officially voted on by the PMC members during the Release Approval Process

Following the ASF rules, the source packages released must be sufficient for a user to build and test the release provided they have access to the appropriate platform and tools.

Convenience packages

There are other ways of installing and using Airflow. Those are "convenience" methods - they are not "official releases" as stated by the ASF Release Policy, but they can be used by the users who do not want to build the software themselves.

Those are - in the order of most common ways people install Airflow:

PyPI releases to install Airflow using standard pip tool
Docker Images to install airflow via docker tool, use them in Kubernetes, Helm Charts, docker-compose, docker swarm, etc. You can read more about using, customising, and extending the images in the Latest docs, and learn details on the internals in the IMAGES.rst document.
Tags in GitHub to retrieve the git project sources that were used to generate official source packages via git

All those artifacts are not official releases, but they are prepared using officially released sources. Some of those artifacts are "development" or "pre-release" ones, and they are clearly marked as such following the ASF Policy.

User Interface

DAGs: Overview of all DAGs in your environment.
Tree: Tree representation of a DAG that spans across time.
Graph: Visualization of a DAG's dependencies and their current status for a specific run.
Task Duration: Total time spent on different tasks over time.
Gantt: Duration and overlap of a DAG.
Code: Quick way to view source code of a DAG.

Semantic versioning

As of Airflow 2.0.0, we support a strict SemVer approach for all packages released.

There are few specific rules that we agreed to that define details of versioning of the different packages:

Airflow: SemVer rules apply to core airflow only (excludes any changes to providers). Changing limits for versions of Airflow dependencies is not a breaking change on its own.
Airflow Providers: SemVer rules apply to changes in the particular provider's code only. SemVer MAJOR and MINOR versions for the packages are independent of the Airflow version. For example, google 4.1.0 and amazon 3.0.3 providers can happily be installed with Airflow 2.1.2. If there are limits of cross-dependencies between providers and Airflow packages, they are present in providers as install_requires limitations. We aim to keep backwards compatibility of providers with all previously released Airflow 2 versions but there will sometimes be breaking changes that might make some, or all providers, have minimum Airflow version specified. Change of that minimum supported Airflow version is a breaking change for provider because installing the new provider might automatically upgrade Airflow (which might be an undesired side effect of upgrading provider).
Airflow Helm Chart: SemVer rules apply to changes in the chart only. SemVer MAJOR and MINOR versions for the chart are independent from the Airflow version. We aim to keep backwards compatibility of the Helm Chart with all released Airflow 2 versions, but some new features might only work starting from specific Airflow releases. We might however limit the Helm Chart to depend on minimal Airflow version.
Airflow API clients: SemVer MAJOR and MINOR versions follow MAJOR and MINOR versions of Airflow. The first MAJOR or MINOR X.Y.0 release of Airflow should always be followed by X.Y.0 release of all clients. The clients then can release their own PATCH releases with bugfixes, independently of Airflow PATCH releases.

Version Life Cycle

Apache Airflow version life cycle:

Version	Current Patch/Minor	State	First Release	Limited Support	EOL/Terminated
2	2.2.2	Supported	Dec 17, 2020	TBD	TBD
1.10	1.10.15	EOL	Aug 27, 2018	Dec 17, 2020	June 17, 2021
1.9	1.9.0	EOL	Jan 03, 2018	Aug 27, 2018	Aug 27, 2018
1.8	1.8.2	EOL	Mar 19, 2017	Jan 03, 2018	Jan 03, 2018
1.7	1.7.1.2	EOL	Mar 28, 2016	Mar 19, 2017	Mar 19, 2017

Limited support versions will be supported with security and critical bug fix only. EOL versions will not get any fixes nor support. We always recommend that all users run the latest available minor release for whatever major version is in use. We highly recommend upgrading to the latest Airflow major release at the earliest convenient time and before the EOL date.

Support for Python and Kubernetes versions

As of Airflow 2.0, we agreed to certain rules we follow for Python and Kubernetes support. They are based on the official release schedule of Python and Kubernetes, nicely summarized in the Python Developer's Guide and Kubernetes version skew policy.

We drop support for Python and Kubernetes versions when they reach EOL. We drop support for those EOL versions in main right after EOL date, and it is effectively removed when we release the first new MINOR (Or MAJOR if there is no new MINOR version) of Airflow For example, for Python 3.6 it means that we drop support in main right after 23.12.2021, and the first MAJOR or MINOR version of Airflow released after will not have it.
The "oldest" supported version of Python/Kubernetes is the default one until we decide to switch to later version. "Default" is only meaningful in terms of "smoke tests" in CI PRs, which are run using this default version and the default reference image available. Currently apache/airflow:latest and apache/airflow:2.2.1 images are Python 3.7 images as we are preparing for 23.12.2021 when will Python 3.6 reaches end of life.
We support a new version of Python/Kubernetes in main after they are officially released, as soon as we make them work in our CI pipeline (which might not be immediate due to dependencies catching up with new versions of Python mostly) we release new images/support in Airflow based on the working CI setup.

Additional notes on Python version requirements

Previous versions require at least Python 3.5.3 when using Python 3.

Contributing

Want to help build Apache Airflow? Check out our contributing documentation.

Official Docker (container) images for Apache Airflow are described in IMAGES.rst.

Who uses Apache Airflow?

More than 400 organizations are using Apache Airflow in the wild.

Who Maintains Apache Airflow?

Airflow is the work of the community, but the core committers/maintainers are responsible for reviewing and merging PRs as well as steering conversations around new feature requests. If you would like to become a maintainer, please review the Apache Airflow committer requirements.

Can I use the Apache Airflow logo in my presentation?

Yes! Be sure to abide by the Apache Foundation trademark policies and the Apache Airflow Brandbook. The most up to date logos are found in this repo and on the Apache Software Foundation website.

Airflow merchandise

If you would love to have Apache Airflow stickers, t-shirt, etc. then check out Redbubble Shop.

Sponsors

The CI infrastructure for Apache Airflow has been sponsored by:

Comments

Re-enable MyPy
Why Mypy re-enable

For a few weeks MyPy checks have been disabled after the switch to Python 3.7 (per https://github.com/apache/airflow/pull/19317).

We should, however, re-enable it back as it is very useful in catching a number of mistakes.

How does it work

We 've re-added the mypy pre-commit now - with mypy bumped to 0.910. This version detects far more errors and we should fix them all before we switch the CI check back.

mypy will be running for incremental changes in pre-commit, same as before. This will enable incremental fixes of the code changed by committers who use pre-commits locally

mypy on CI runs in non-failing mode. When the main pre-commit check is run, mypy is disabled, but then it is run as a separate step (which does not fail but will show the result of running mypy on all our code). This will enable us to track the progress of fixes

Can I help with the effort, you ask?

We started concerted effort now and incrementally fix all the mypy incompatibilities - ideally package/by/package to avoid huge code reviews. We'd really appreciate a number of people to contribute, so that we can re-enable mypy back fully and quickly :).

How can I help?

What you need is:

checkout main

./breeeze build-image

pip install pre-commit

pre-commit install

This will enable automated checks for when you do a regular contribution. When you make your change, any MyPy issues will be reporteed and you need to fix them all to commit. You can also commit with --no-verify flag to skip that, bu, well, if you can improve airlfow a little - why not?

How can I help more ?

You can add PRs that are fixing whole packages, without contributing features or bugfixes. Please refer to this issue #19891 and ideally comment below in the issue that you want to take care of a package (to avoid duplicate work).

An easy way to run MyPy check for package can be done either from the host:

find DIRECTORY -name "*.py" | xargs pre-commit run mypy --files

or from ./breeze shell:

mypy --namespace-packages DIRECTORY

Current list of mypy PRs:

https://github.com/apache/airflow/pulls?q=is%3Aopen+is%3Apr+label%3Amypy

Remaining packages

Here is the list of remaining packages to be "mypy compliant" generated with:

pre-commit run mypy --all-files 2>&1 | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" | grep "error:" | sort | awk 'FS=":" { print $1 }' | xargs dirname | sort | uniq -c | xargs -n 2 printf "* [ ] (%4d) %s\n"

[ ] ( 1) airflow/api/common/experimental

[ ] ( 1) airflow/contrib/sensors

[ ] ( 1) airflow/example_dags

[ ] ( 1) airflow/jobs

[ ] ( 4) airflow/models

[ ] ( 1) airflow/providers/microsoft/winrm/hooks

[ ] ( 1) airflow/providers/ssh/hooks

[ ] ( 1) tests/providers/amazon/aws/hooks

[ ] ( 1) tests/providers/google/cloud/hooks

Committer

[X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

area:dev-env kind:meta mypy
opened by potiuk 157
Task stuck in "scheduled" or "queued" state, pool has all slots queued, nothing is executing
Apache Airflow version: 2.0.0

Kubernetes version (if you are using kubernetes) (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.14-gke.1600", GitCommit:"7c407f5cc8632f9af5a2657f220963aa7f1c46e7", GitTreeState:"clean", BuildDate:"2020-12-07T09:22:27Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: GKE

OS (e.g. from /etc/os-release):

Kernel (e.g. uname -a):

Install tools:

Others:

Airflow metadata database is hooked up to a PostgreSQL instance

What happened:

Airflow 2.0.0 running on the KubernetesExecutor has many tasks stuck in "scheduled" or "queued" state which never get resolved.

The setup has a default_pool of 16 slots.

Currently no slots are used (see Screenshot), but all slots are queued.

No work is executed any more. The Executor or Scheduler is stuck.

There are many many tasks stuck in "scheduled" state

Tasks in "scheduled" state say ('Not scheduling since there are %s open slots in pool %s and require %s pool slots', 0, 'default_pool', 1) That is simply not true, because there is nothing running on the cluster and there are always 16 tasks stuck in "queued".

There are many tasks stuck in "queued" state

Tasks in "queued" state say Task is in the 'running' state which is not a valid state for execution. The task must be cleared in order to be run. That is also not true. Nothing is running on the cluster and Airflow is likely just lying to itself. It seems the KubernetesExecutor and the scheduler easily go out of sync.

What you expected to happen:

Airflow should resolve scheduled or queued tasks by itself once the pool has available slots

Airflow should use all available slots in the pool

It should be possible to clear a couple hundred tasks and expect the system to stay consistent

How to reproduce it:

Vanilla Airflow 2.0.0 with KubernetesExecutor on Python 3.7.9

requirements.txt

pyodbc==4.0.30 pycryptodomex==3.9.9 apache-airflow-providers-google==1.0.0 apache-airflow-providers-odbc==1.0.0 apache-airflow-providers-postgres==1.0.0 apache-airflow-providers-cncf-kubernetes==1.0.0 apache-airflow-providers-sftp==1.0.0 apache-airflow-providers-ssh==1.0.0

The only reliable way to trigger that weird bug is to clear the task state of many tasks at once. (> 300 tasks)

Anything else we need to know:

Don't know, as always I am happy to help debug this problem. The scheduler/executer seems to go out of sync and never back in sync again with the state of the world.

We actually planned to upscale our Airflow installation with many more simultaneous tasks. With these severe yet basic scheduling/queuing problems we cannot move forward at all.

Another strange, likely unrelated observation, the scheduler always uses 100% of the CPU. Burning it. Even with no scheduled or now queued tasks, its always very very busy.

Workaround:

The only workaround for this problem I could find so far, is to manually go in, find all tasks in "queued" state and clear them all at once. Without that, the whole cluster/Airflow just stays stuck like it is.
kind:bug priority:high affected_version:2.0
opened by Overbryd 145
Add Compute Engine SSH hook
Closes #9752

[x] Add Compute Engine SSH Hook and Operator

[x] Add Example DAG and Documentation

[x] Add System tests

Make sure to mark the boxes below before creating PR: [x]

[x] Description above provides context of the change

[x] Unit tests coverage for changes (not needed for documentation changes)

[x] Target Github ISSUE in description if exists

[x] Commits follow "How to write a good git commit message"

[x] Relevant documentation is updated including usage instructions.

[x] I will engage committers as explained in Contribution Workflow Example.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in UPDATING.md. Read the Pull Request Guidelines for more information.
provider:Google pinned full tests needed
opened by ephraimbuddy 102
Added windows extensions

This PR was created after a discussion in this post. @potiuk asked to be mentioned here so he can work with us to integrate those gradual changes.

The aim of this PR is to gradually enable support for the Windows Platform. The changes in this PR allow Airflow to be used with on a Windows System as Dask-Worker. Other things like Web-Server and Task-Scheduling are not possible because of the way processes are handled in Airflow. Next steps would be to find suitable alternatives to those POSIX process management concepts that work on Windows.
area:scheduler/executor area:logging type:new-feature area:core-operators

opened by casra-developers 87
[WIP] Airflow kubernetes executor
Dear Airflow maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

[x] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-1314

Description

[x] Here are some details about my PR, including screenshots of any UI changes: A full proposal for this PR can be found here https://github.com/bloomberg/airflow/blob/29694ae9903c4dad3f18fb8eb767c4922dbef2e8/dimberman-KubernetesExecutorProposal-050717-1423-36.pdf

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line

Subject is limited to 50 characters

Subject does not end with a period

Subject uses the imperative mood ("add", not "adding")

Body wraps at 72 characters

Body explains "what" and "why", not "how"
opened by dimberman 83
enter the shell breeze2 environment

closes: #20960 related: #20960 Breeze2 - enter the shell in breeze2

^ Add meaningful description above

Read the Pull Request Guidelines for more information. In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in UPDATING.md.
area:dev-tools

opened by Bowrna 81
[AIRFLOW-4030] second attempt to add singularity to airflow
This is a second attempt to add Singularity Container support to Apache Airflow by way of Singularity Python. I am using the previously created JIRA ticket 4030 (created in March 2019) as it is still relevant. I am a new contributor and largely not familiar with the community here (and yes I've read the guidelines) so I would appreciate support and kindness from the individuals that act as maintainers here, and any additional support from other folks that are also interested in this integration. Thank you!

Signed-off-by: Vanessa Sochat [email protected]

Issue link: AIRFLOW-4030

Make sure to mark the boxes below before creating PR: [x]

[x] Description above provides context of the change

[x] Commit message/PR title starts with [AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID^*

[x] Unit tests coverage for changes (not needed for documentation changes)

[x] Commits follow "How to write a good git commit message"

[x] Relevant documentation is updated including usage instructions.

[x] I will engage committers as explained in Contribution Workflow Example.

^* For document-only changes commit message can start with [AIRFLOW-XXXX].

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in UPDATING.md. Read the Pull Request Guidelines for more information.
area:dev-tools
opened by vsoch 78
[AIRFLOW-1424] make the next execution date of DAGs visible
Dear Airflow maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

[x] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-1424

Description

[x] Here are some details about my PR, including screenshots of any UI changes:

The scheduler's DAG run creation logic can be tricky and one is easily confused with the start_date + interval and period end scheduling way of thinking.

It would ease airflow's usage to add a next execution field to DAGs so that we can very easily see the (un)famous period end after which the scheduler will create a new DAG run for our workflows.

These patches are a simple way to implement this on the DAG model and make use of this in the interface.

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Tests are provided

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line

Subject is limited to 50 characters

Subject does not end with a period

Subject uses the imperative mood ("add", not "adding")

Body wraps at 72 characters

Body explains "what" and "why", not "how"

pinned AIP-39
opened by ultrabug 77
[AIRFLOW-5033] Switched to snakebite-py3
NOTE TO REVIEWER: It depends on #7278 - so please check only the last commit.

Make sure you have checked all steps below.

Jira

[x] My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-5033

Description

[x] Here are some details about my PR, including screenshots of any UI changes: Snakebite is not PY3-compatible. Trying to switch to alledgedly drop-in replacement which is py3-compatible.

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line

Subject is limited to 50 characters (not including Jira issue reference)

Subject does not end with a period

Subject uses the imperative mood ("add", not "adding")

Body wraps at 72 characters

Body explains "what" and "why", not "how"

Documentation

[x] In case of new functionality, my PR adds documentation that describes how to use it.

All the public functions and the classes in the PR contain docstrings that explain what it does

If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

[x] Passes flake8

Help Wanted area:dev-tools
opened by potiuk 74
Tasks intermittently gets terminated with SIGTERM on kubernetes executor
Apache Airflow version

2.1.3 (latest released)

Operating System

Linux

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

Have tried env variables as given in this github issue issues/14672: AIRFLOW__CORE__KILLED_TASK_CLEANUP_TIME: "604800" AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION: "False"

What happened

[2021-09-04 10:28:50,536] {local_task_job.py:80} ERROR - Received SIGTERM. Terminating subprocesses [2021-09-04 10:28:50,536] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 33 [2021-09-04 10:28:50,537] {taskinstance.py:1235} ERROR - Received SIGTERM. Terminating subprocesses. [2021-09-04 10:28:52,568] {taskinstance.py:1462} ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1164, in _run_raw_task self._prepare_and_execute_task_with_callbacks(context, task) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1282, in _prepare_and_execute_task_with_callbacks result = self._execute_task(context, task_copy) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1307, in _execute_task result = task_copy.execute(context=context) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 150, in execute return_value = self.execute_callable() File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 161, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/opt/airflow/dags/repo/dags/elastit_schedular/waiting_task_processor.py", line 59, in trigger_task time.sleep(1) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1237, in signal_handler raise AirflowException("Task received SIGTERM signal") airflow.exceptions.AirflowException: Task received SIGTERM signal

What you expected to happen

Dag must get executed successfully without any sigterm signal.

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

[x] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

kind:bug area:core
opened by Nimesh-K-Makwana 71
scheduler gets stuck without a trace
Apache Airflow version:

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

Cloud provider or hardware configuration:

OS (e.g. from /etc/os-release):

Kernel (e.g. uname -a):

Install tools:

Others: What happened:

The scheduler gets stuck without a trace or error. When this happens, the CPU usage of scheduler service is at 100%. No jobs get submitted and everything comes to a halt. Looks it goes into some kind of infinite loop. The only way I could make it run again is by manually restarting the scheduler service. But again, after running some tasks it gets stuck. I've tried with both Celery and Local executors but same issue occurs. I am using the -n 3 parameter while starting scheduler.

Scheduler configs, job_heartbeat_sec = 5 scheduler_heartbeat_sec = 5 executor = LocalExecutor parallelism = 32

Please help. I would be happy to provide any other information needed

What you expected to happen:

How to reproduce it:

Anything else we need to know:

Moved here from https://issues.apache.org/jira/browse/AIRFLOW-401
kind:bug area:scheduler/executor
opened by dimberman 71
Handle non-compliant behaviour of Exasol cursor

Exasol is not DBAPI-2 compliant, as described in: https://github.com/exasol/pyexasol/blob/master/docs/DBAPI_COMPAT.md

This means that SQLOperator cannot be used for it and we should undeprecate the original operator and use specific Exasol handler as default for the returned values.

We also add explicit exception in case non-compliant DB2-API database (with missing description) is used with default Operator and handlers.

Fixes: #28731

^ Add meaningful description above

Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
area:providers provider:Common-sql

opened by potiuk 1
s3_hook.copy_object() doesn't allow copying files > 5Gb
Apache Airflow version

2.5.0

What happened

I have a DAG in which I copy files from one bucket to another. Today, a call to s3Hook broke because s3_hook.copy_object() can't handle more than 5 Gb, and I have a parquet file of 6Gb. My temporal fix was moving it manually, but I thought a permanent fix would be using s3_hook.copy instead of s3_hook.copy_object, the problem is that it doesn't exist.

In my mind they do the same thing, so if we change in https://github.com/apache/airflow/blob/6cbf9b62e374666693b93f79a36dcb46cc8245c4/airflow/providers/amazon/aws/hooks/s3.py#L814 copy_object to copy wouldn't do the same and also move files bigger than 5 Gb? or there is some other method for this case? The documentation for copy() is in here.

What you think should happen instead

I think it shouldn't matter the size of the file while copying files. In my case, the log it showed was:

Traceback (most recent call last): File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/decorators/base.py", line 188, in execute return_value = super().execute(context) File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/operators/python.py", line 175, in execute return_value = self.execute_callable() File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/operators/python.py", line 193, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/opt/c2/airflow/c1/dags/dataengineers/c3/create_dev_env.py", line 32, in copy_input_data _copy_input_data( File "/opt/c2/airflow/c1/dags/dataengineers/c3/actions_create.py", line 155, in _copy_input_data _copy_s3_data(source_bucket, entity["storage"], dest_bucket, folder_prefix) File "/opt/c2/airflow/c1/dags/dataengineers/c3/actions_create.py", line 87, in _copy_s3_data s3_hook.copy_object( File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 830, in copy_object response = self.get_conn().copy_object( File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/botocore/client.py", line 508, in _api_call return self._make_api_call(operation_name, kwargs) File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/botocore/client.py", line 915, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the CopyObject operation: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120

How to reproduce

No response

Operating System

ubuntu 20.04

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==6.0.0 apache-airflow-providers-cncf-kubernetes==4.4.0 apache-airflow-providers-common-sql==1.2.0 apache-airflow-providers-ftp==3.1.0 apache-airflow-providers-google==6.8.0 apache-airflow-providers-http==4.0.0 apache-airflow-providers-imap==3.0.0 apache-airflow-providers-postgres==5.2.2 apache-airflow-providers-sendgrid==3.0.0 apache-airflow-providers-slack==6.0.0 apache-airflow-providers-sqlite==3.2.1

Deployment

Other 3rd-party Helm chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

kind:bug area:core
opened by set92 1
Fix serialize() usages in custom JSON encoders

Fix #28741.

Our custom JSON encoders call serialize() in their default() hooks. However, serialize() does not guarantee to return a JSON-serializable type, and when it does not (e.g. a set object), the JSON encoder would infinitely recurse into serialize() trying to coerce the object into JSON-serializable.

This patch adds an additional ensure_json_compatible parameter to serialize(), so JSON encoders can use it to require serialize() to always return JSON-compatible values.

Some minor refactoring is also applied to serialize() to ensure the flag is correctly passed down through recursive visits, and also hide the depth tracking parameter (which is internal to the function) from outside access.

The _reverse_cache global variable is also deleted since it is not used anywhere, even before this patch.
area:serialization

opened by uranusjr 0
RecursionError with new refactored serialization
Apache Airflow version

main (development)

What happened

>>> from airflow.utils.json import XComEncoder >>> XComEncoder().encode(set()) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/airflow/airflow/utils/json.py", line 91, in encode return super().encode(o) File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/opt/airflow/airflow/utils/json.py", line 82, in default return serialize(o) File "/opt/airflow/airflow/serialization/serde.py", line 97, in serialize if isinstance(o, _primitives): RecursionError: maximum recursion depth exceeded in __instancecheck__

What you think should happen instead

The encoder should correctly encode a value, or raise TypeError if it is not encodable.

How to reproduce

Simply use XComEncoder to encode anything that’s not encodable by the built-in JSONEncoder (i.e. something that’d go through the custom default hook).

Operating System

Any

Versions of Apache Airflow Providers

Irrelevant

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

The problem here is because serialize does not guarantee to return a JSON-serializable object. So when default receives an set, it returns a set, and the encoder would try to serialize that set again, calling default, and eventually the stack explodes.

The simplest solution would be to add an additional flag to serialize so it can return a JSON-compatible type if requested.

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

kind:bug area:core
opened by uranusjr 1
airflow workers and scheduler memory leak
Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

In airflow 2.2.5, The worker and scheduler memory gradually increasing day by day when no tasks are running. The log rotation also enabled. We have enabled the celery inspect command with liveness check probe for workers to resolve the communication issue between worker and airflow redis.

Liveness: exec [sh -c CONNECTION_CHECK_MAX_COUNT=0 exec /entrypoint python -m celery --app airflow.executors.celery_executor.app inspect ping -d celery@${HOSTNAME}] delay=30s timeout=60s period=30s #success=1 #failure=5

I have disabled the liveness check and didn't observe the memory leak. how to enable the liveness checks to resolve disconnection between redis and workers? why redis and workers communication not resuming automatically.

I observed memory leak issues there for scheduler pod also. but didn't find the root cause.

What you think should happen instead

why celery inspect command causing the memory leak in worker.

The redis and worker communication should resume automatically without any intervention.

The scheduler memory leak should not happen. The scheduler utilization go up while processing tasks and it has to comedown once done.

How to reproduce

Install airflow 2.2.5 with Python 3.8 to replicate the issues.

for workers memory leak: enable the below command and observe the memory utilization.

Liveness: exec [sh -c CONNECTION_CHECK_MAX_COUNT=0 exec /entrypoint python -m celery --app airflow.executors.celery_executor.app inspect ping -d celery@${HOSTNAME}] delay=30s timeout=60s period=30s #success=1 #failure=5

The scheduler no need of any changes. The memory gradually increasing day by day.

Operating System

PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

Versions of Apache Airflow Providers

airflow 2.2.5

Deployment

Official Apache Airflow Helm Chart

Deployment details

NA

Anything else

No response

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

kind:bug area:core
opened by anu251989 1
Add --overwrite option to `connections import` CLI command

The CLI command airflow connections import <filepath> cannot overwrite existing connections. Conflicts are skipped with the message Could not import connection <connection_id>: connection already exists.

This PR adds functionality to set --overwrite=true to overwrite existing connections when there is a conflict.

The existing behavior (no overwrites) is still the default behavior, the user must set --overwrite=true to enable overwrites.

Original idea for this: https://github.com/apache/airflow/pull/15177#issuecomment-1286555955
area:CLI

opened by natanweinberger 0

Releases(2.5.0)

2.5.0(Dec 2, 2022)
New Features

TaskRunner: notify of component start and finish (#27855)

Add DagRun state change to the Listener plugin system(#27113)

Metric for raw task return codes (#27155)

Add logic for XComArg to pull specific map indexes (#27771)

Clear TaskGroup (#26658, #28003)

Add critical section query duration metric (#27700)

Add: #23880 :: Audit log for AirflowModelViews(Variables/Connection) (#24079, #27994, #27923)

Add postgres 15 support (#27444)

Expand tasks in mapped group at run time (#27491)

reset commits, clean submodules (#27560)

scheduler_job, add metric for scheduler loop timer (#27605)

Allow datasets to be used in taskflow (#27540)

Add expanded_ti_count to ti context (#27680)

Add user comment to task instance and dag run (#26457, #27849, #27867)

Enable copying DagRun JSON to clipboard (#27639)

Implement extra controls for SLAs (#27557)

add dag parsed time in DAG view (#27573)

Add max_wait for exponential_backoff in BaseSensor (#27597)

Expand tasks in mapped group at parse time (#27158)

Add disable retry flag on backfill (#23829)

Adding sensor decorator (#22562)

Api endpoint update ti (#26165)

Filtering datasets by recent update events (#26942)

Support Is /not Null filter for value is None on webui (#26584)

Add search to datasets list (#26893)

Split out and handle 'params' in mapped operator (#26100)

Add authoring API for TaskGroup mapping (#26844)

Add one_done trigger rule (#26146)

Create a more efficient airflow dag test command that also has better local logging (#26400)

Support add/remove permissions to roles commands (#26338)

Auto tail file logs in Web UI (#26169)

Add triggerer info to task instance in API (#26249)

Flag to deserialize value on custom XCom backend (#26343)

Improvements

Allow depth-first execution (#27827)

UI: Update offset height if data changes (#27865)

Improve TriggerRuleDep typing and readability (#27810)

Make views requiring session, keyword only args (#27790)

Optimize TI.xcom_pull() with explicit task_ids and map_indexes (#27699)

Allow hyphens in pod id used by k8s executor (#27737)

optimise task instances filtering (#27102)

Use context managers to simplify log serve management (#27756)

Fix formatting leftovers (#27750)

Improve task deadlock messaging (#27734)

Improve "sensor timeout" messaging (#27733)

Replace urlparse with urlsplit (#27389)

Align TaskGroup semantics to AbstractOperator (#27723)

Add new files to parsing queue on every loop of dag processing (#27060)

Make Kubernetes Executor & Scheduler resilient to error during PMH execution (#27611)

Separate dataset deps into individual graphs (#27356)

Use log.exception where more economical than log.error (#27517)

Move validation branch_task_ids into SkipMixin (#27434)

Coerce LazyXComAccess to list when pushed to XCom (#27251)

Update cluster-policies.rst docs (#27362)

Add warning if connection type already registered within the provider (#27520)

Activate debug logging in commands with --verbose option (#27447)

Add classic examples for Python Operators (#27403)

change .first() to .scalar() (#27323)

Improve reset_dag_run description (#26755)

Add examples and howtos about sensors (#27333)

Make grid view widths adjustable (#27273)

Sorting plugins custom menu links by category before name (#27152)

Simplify DagRun.verify_integrity (#26894)

Add mapped task group info to serialization (#27027)

Correct the JSON style used for Run config in Grid View (#27119)

No extra__conn_type__ prefix required for UI behaviors (#26995)

Improve dataset update blurb (#26878)

Rename kubernetes config section to kubernetes_executor (#26873)

decode params for dataset searches (#26941)

Get rid of the DAGRun details page & rely completely on Grid (#26837)

Fix scheduler crashloopbackoff when using hostname_callable (#24999)

Reduce log verbosity in KubernetesExecutor. (#26582)

Don't iterate tis list twice for no reason (#26740)

Clearer code for PodGenerator.deserialize_model_file (#26641)

Don't import kubernetes unless you have a V1Pod (#26496)

Add updated_at column to DagRun and Ti tables (#26252)

Move the deserialization of custom XCom Backend to 2.4.0 (#26392)

Avoid calculating all elements when one item is needed (#26377)

Add __future__.annotations automatically by isort (#26383)

Handle list when serializing expand_kwargs (#26369)

Apply PEP-563 (Postponed Evaluation of Annotations) to core airflow (#26290)

Add more weekday operator and sensor examples #26071 (#26098)

Align TaskGroup semantics to AbstractOperator (#27723)

Bug Fixes

Gracefully handle whole config sections being renamed (#28008)

Add allow list for imports during deserialization (#27887)

Soft delete datasets that are no longer referenced in DAG schedules or task outlets (#27828)

Redirect to home view when there are no valid tags in the URL (#25715)

Refresh next run datasets info in dags view (#27839)

Make MappedTaskGroup depend on its expand inputs (#27876)

Make DagRun state updates for paused DAGs faster (#27725)

Don't explicitly set include_examples to False on task run command (#27813)

Fix menu border color (#27789)

Fix backfill queued task getting reset to scheduled state. (#23720)

Fix clearing child dag mapped tasks from parent dag (#27501)

Handle json encoding of V1Pod in task callback (#27609)

Fix ExternalTaskSensor can't check zipped dag (#27056)

Avoid re-fetching DAG run in TriggerDagRunOperator (#27635)

Continue on exception when retrieving metadata (#27665)

External task sensor fail fix (#27190)

Add the default None when pop actions (#27537)

Display parameter values from serialized dag in trigger dag view. (#27482, #27944)

Move TriggerDagRun conf check to execute (#27035)

Resolve trigger assignment race condition (#27072)

Update google_analytics.html (#27226)

Fix some bug in web ui dags list page (auto-refresh & jump search null state) (#27141)

Fixed broken URL for docker-compose.yaml (#26721)

Fix xcom arg.py .zip bug (#26636)

Fix 404 taskInstance errors and split into two tables (#26575)

Fix browser warning of improper thread usage (#26551)

template rendering issue fix (#26390)

Clear autoregistered DAGs if there are any import errors (#26398)

Fix from airflow import version lazy import (#26239)

allow scroll in triggered dag runs modal (#27965)

Misc/Internal

Remove is_mapped attribute (#27881)

Simplify FAB table resetting (#27869)

Fix old-style typing in Base Sensor (#27871)

Switch (back) to late imports (#27730)

Completed D400 for multiple folders (#27748)

simplify notes accordion test (#27757)

completed D400 for airflow/callbacks/* airflow/cli/* (#27721)

Completed D400 for airflow/api_connexion/* directory (#27718)

Completed D400 for airflow/listener/* directory (#27731)

Completed D400 for airflow/lineage/* directory (#27732)

Update API & Python Client versions (#27642)

Completed D400 & D401 for airflow/api/* directory (#27716)

Completed D400 for multiple folders (#27722)

Bump minimatch from 3.0.4 to 3.0.8 in /airflow/www (#27688)

Bump loader-utils from 1.4.1 to 1.4.2in /airflow/www (#27697)

Disable nested task mapping for now (#27681)

bump alembic minimum version (#27629)

remove unused code.html (#27585)

Enable python string normalization everywhere (#27588)

Upgrade dependencies in order to avoid backtracking (#27531)

Strengthen a bit and clarify importance of triaging issues (#27262)

Deduplicate type hints (#27508)

Add stub 'yield' to BaseTrigger.run (#27416)

Remove upper-bound limit to dask (#27415)

Limit Dask to under 2022.10.1 (#27383)

Update old style typing (#26872)

Enable string normalization for docs (#27269)

Slightly faster up/downgrade tests (#26939)

Deprecate use of core get_kube_client in PodManager (#26848)

Add memray files to gitignore / dockerignore (#27001)

Bump sphinx and sphinx-autoapi (#26743)

Simplify RTIF.delete_old_records() (#26667)

migrate last react files to typescript (#26112)

Work around pyupgrade edge cases (#26384)

Doc only changes

Document dag_file_processor_timeouts metric as deprecated (#27067)

Drop support for PostgreSQL 10 (#27594)

Update index.rst (#27529)

Add note about pushing the lazy XCom proxy to XCom (#27250)

Fix BaseOperator link (#27441)

[docs] best-practices add use variable with template example. (#27316)

docs for custom view using plugin (#27244)

Update graph view and grid view on overview page (#26909)

Documentation fixes (#26819)

make consistency on markup title string level (#26696)

Add documentation to dag test function (#26713)

Fix broken URL for docker-compose.yaml (#26726)

Add a note against use of top level code in timetable (#26649)

Fix example_datasets dag names (#26495)

Update docs: zip-like effect is now possible in task mapping (#26435)

changing to task decorator in docs from classic operator use (#25711)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.5.0-source.tar.gz(22.15 MB)
apache-airflow-2.5.0-source.tar.gz.asc(265 bytes)
apache-airflow-2.5.0-source.tar.gz.sha512(165 bytes)
apache-airflow-2.5.0.tar.gz(5.79 MB)
apache-airflow-2.5.0.tar.gz.asc(265 bytes)
apache-airflow-2.5.0.tar.gz.sha512(158 bytes)
apache_airflow-2.5.0-py3-none-any.whl(6.32 MB)
apache_airflow-2.5.0-py3-none-any.whl.asc(265 bytes)
apache_airflow-2.5.0-py3-none-any.whl.sha512(168 bytes)
2.4.3(Nov 14, 2022)
Bug Fixes

Fix double logging with some task logging handler (#27591)

Replace FAB url filtering function with Airflow's (#27576)

Fix mini scheduler expansion of mapped task (#27506)

SLAMiss is nullable and not always given back when pulling task instances (#27423)

Fix behavior of _ when searching for DAGs (#27448)

Fix getting the dag/task ids from BaseExecutor (#27550)

Fix SQLAlchemy primary key black-out error on DDRQ (#27538)

Fix IntegrityError during webserver startup (#27297)

Add case insensitive constraint to username (#27266)

Fix python external template keys (#27256)

Reduce extraneous task log requests (#27233)

Make RotatingFilehandler used in DagProcessor non-caching (#27223)

Listener: Set task on SQLAlchemy TaskInstance object (#27167)

Fix dags list page auto-refresh & jump search null state (#27141)

Set executor.job_id to BackfillJob.id for backfills (#27020)

Misc/Internal

Bump loader-utils from 1.4.0 to 1.4.1 in /airflow/www (#27552)

Reduce log level for k8s TCP_KEEPALIVE etc warnings (#26981)

Doc only changes

Use correct executable in docker compose docs (#27529)

Fix wording in DAG Runs description (#27470)

Document that KubernetesExecutor overwrites container args (#27450)

Fix BaseOperator links (#27441)

Correct timer units to seconds from milliseconds. (#27360)

Add missed import in the Trigger Rules example (#27309)

Update SLA wording to reflect it is relative to Dag Run start. (#27111)

Add kerberos environment variables to the docs (#27028)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.4.3-source.tar.gz(21.96 MB)
apache-airflow-2.4.3-source.tar.gz.asc(265 bytes)
apache-airflow-2.4.3-source.tar.gz.sha512(165 bytes)
apache-airflow-2.4.3.tar.gz(5.69 MB)
apache-airflow-2.4.3.tar.gz.asc(265 bytes)
apache-airflow-2.4.3.tar.gz.sha512(158 bytes)
apache_airflow-2.4.3-py3-none-any.whl(6.20 MB)
apache_airflow-2.4.3-py3-none-any.whl.asc(265 bytes)
apache_airflow-2.4.3-py3-none-any.whl.sha512(168 bytes)
2.4.2(Oct 24, 2022)
Bug Fixes

Make tracebacks opt-in (#27059)

Add missing AUTOINC/SERIAL for FAB tables (#26885)

Add separate error handler for 405(Method not allowed) errors (#26880)

Don't re-patch pods that are already controlled by current worker (#26778)

Handle mapped tasks in task duration chart (#26722)

Fix task duration cumulative chart (#26717)

Avoid 500 on dag redirect (#27064)

Filter dataset dependency data on webserver (#27046)

Remove double collection of dags in airflow dags reserialize (#27030)

Fix auto refresh for graph view (#26926)

Don't overwrite connection extra with invalid json (#27142)

Fix next run dataset modal links (#26897)

Change dag audit log sort by date from asc to desc (#26895)

Bump min version of jinja2 (#26866)

Add missing colors to state_color_mapping jinja global (#26822)

Fix running debuggers inside airflow tasks test (#26806)

Fix warning when using xcomarg dependencies (#26801)

demote Removed state in priority for displaying task summaries (#26789)

Ensure the log messages from operators during parsing go somewhere (#26779)

Add restarting state to TaskState Enum in REST API (#26776)

Allow retrieving error message from data.detail (#26762)

Simplify origin string cleaning (#27143)

Remove DAG parsing from StandardTaskRunner (#26750)

Fix non-hidden cumulative chart on duration view (#26716)

Remove TaskFail duplicates check (#26714)

Fix airflow tasks run --local when dags_folder differs from that of processor (#26509)

Fix yarn warning from d3-color (#27139)

Fix version for a couple configurations (#26491)

Revert "No grid auto-refresh for backfill dag runs (#25042)" (#26463)

Retry on Airflow Schedule DAG Run DB Deadlock (#26347)

Misc/Internal

Clean-ups around task-mapping code (#26879)

Move user-facing string to template (#26815)

add icon legend to datasets graph (#26781)

Bump sphinx and sphinx-autoapi (#26743)

Simplify RTIF.delete_old_records() (#26667)

Bump FAB to 4.1.4 (#26393)

Doc only changes

Fixed triple quotes in task group example (#26829)

Documentation fixes (#26819)

make consistency on markup title string level (#26696)

Add a note against use of top level code in timetable (#26649)

Fix broken URL for docker-compose.yaml (#26726)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.4.2-source.tar.gz(21.96 MB)
apache-airflow-2.4.2-source.tar.gz.asc(265 bytes)
apache-airflow-2.4.2-source.tar.gz.sha512(165 bytes)
apache-airflow-2.4.2.tar.gz(5.68 MB)
apache-airflow-2.4.2.tar.gz.asc(265 bytes)
apache-airflow-2.4.2.tar.gz.sha512(158 bytes)
apache_airflow-2.4.2-py3-none-any.whl(6.19 MB)
apache_airflow-2.4.2-py3-none-any.whl.asc(265 bytes)
apache_airflow-2.4.2-py3-none-any.whl.sha512(168 bytes)
helm-chart/1.7.0(Oct 14, 2022)
New Features

Make cleanup job history configurable (#26838)

Added labels to specific Airflow components (#25031)

Add StatsD overrideMappings in Helm chart values (#26598)

Adding podAnnotations to StatsD deployment template (#25732)

Container specific extra environment variables (#24784)

Custom labels for extra Secrets and ConfigMaps (#25283)

Add revisionHistoryLimit to all deployments (#25059)

Adding podAnnotations to Redis StatefulSet (#23708)

Provision Standalone Dag Processor (#23711)

Add configurable scheme for webserver probes (#22815)

Add support for KEDA HPA config to Helm chart (#24220)

Improvements

Add 'executor' label to Airflow scheduler deployment (#25684)

Add default flower_url_prefix in Helm chart values (#26415)

Add liveness probe to Celery workers (#25561)

Use sql_alchemy_conn for celery result backend when result_backend is not set (#24496)

Bug Fixes

Fix pod template imagePullPolicy (#26423)

Do not declare a volume for sshKeySecret if dag persistence is enabled (#22913)

Pass worker annotations to generated pod template (#24647)

Fix semver compare number for jobs check command (#24480)

Use --local flag for liveness probes in Airflow 2.5+ (#24999)

Doc only changes

Improve documentation on helm hooks disabling (#26747)

Remove ssh:// prefix from git repo value (#26632)

Fix defaultAirflowRepository comment (#26428)

Baking DAGs into Docker image (#26401)

Reload pods when using the same DAG tag (#24576)

Minor clarifications about result_backend, dag processor, and helm uninstall (#24929)

Add hyperlinks to GitHub PRs for Release Notes (#24532)

Terraform should not use Helm hooks for starting jobs (#26604)

Flux should not use Helm hooks for starting jobs (#24288)

Provide details on how to pull Airflow image from a private repository (#24394)

Helm logo no longer a link (#23977)

Document LocalKubernetesExecutor support in chart (#23876)

Update Production Guide (#23836)

Misc

Default Airflow version to 2.4.1 (#26485)

Vendor in the Bitnami chart (#24395)

Remove kubernetes 1.20 support (#25871)

Source code(tar.gz)
Source code(zip)
airflow-1.7.0.tgz(153.37 KB)
airflow-1.7.0.tgz.asc(870 bytes)
airflow-1.7.0.tgz.prov(9.31 KB)
airflow-1.7.0.tgz.sha512(148 bytes)
airflow-chart-1.7.0-source.tar.gz(157.58 KB)
airflow-chart-1.7.0-source.tar.gz.asc(870 bytes)
airflow-chart-1.7.0-source.tar.gz.sha512(164 bytes)
2.4.1(Sep 30, 2022)
Bug Fixes

When rendering template, unmap task in context (#26702)

Fix scroll overflow for ConfirmDialog (#26681)

Resolve deprecation warning re Table.exists() (#26616)

Fix XComArg zip bug (#26636)

Use COALESCE when ordering runs to handle NULL (#26626)

Check user is active (#26635)

No missing user warning for public admin (#26611)

Allow MapXComArg to resolve after serialization (#26591)

Resolve warning about DISTINCT ON query on dags view (#26608)

Log warning when secret backend kwargs is invalid (#26580)

Fix grid view log try numbers (#26556)

Template rendering issue in passing templates_dict to task decorator (#26390)

Fix Deferrable stuck as scheduled during backfill (#26205)

Suppress SQLALCHEMY_TRACK_MODIFICATIONS warning in db init (#26617)

Correctly set json_provider_class on Flask app so it uses our encoder (#26554)

Fix WSGI root app (#26549)

Fix deadlock when mapped task with removed upstream is rerun (#26518)

ExecutorConfigType should be cacheable (#26498)

Fix proper joining of the path for logs retrieved from celery workers (#26493)

DAG Deps extends base_template (#26439)

Don't update backfill run from the scheduler (#26342)

Doc only changes

Clarify owner links document (#26515)

Fix invalid RST in dataset concepts doc (#26434)

Document the non-sensitive-only option for expose_config (#26507)

Fix example_datasets dag names (#26495)

Zip-like effect is now possible in task mapping (#26435)

Use task decorator in docs instead of classic operators (#25711)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.4.1-source.tar.gz(21.95 MB)
apache-airflow-2.4.1-source.tar.gz.asc(870 bytes)
apache-airflow-2.4.1-source.tar.gz.sha512(165 bytes)
apache-airflow-2.4.1.tar.gz(5.68 MB)
apache-airflow-2.4.1.tar.gz.asc(870 bytes)
apache-airflow-2.4.1.tar.gz.sha512(158 bytes)
apache_airflow-2.4.1-py3-none-any.whl(6.19 MB)
apache_airflow-2.4.1-py3-none-any.whl.asc(870 bytes)
apache_airflow-2.4.1-py3-none-any.whl.sha512(168 bytes)
2.4.0(Sep 19, 2022)
New Features

Add Data-aware Scheduling (https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-48+milestone%3A%22Airflow+2.4.0%22)

Add @task.short_circuit TaskFlow decorator (#25752)

Make execution_date_or_run_id optional in tasks test command (#26114)

Automatically register DAGs that are used in a context manager (#23592, #26398)

Add option of sending DAG parser logs to stdout. (#25754)

Support multiple DagProcessors parsing files from different locations. (#25935)

Implement ExternalPythonOperator (#25780)

Make execution_date optional for command dags test (#26111)

Implement expand_kwargs() against a literal list (#25925)

Add trigger rule tooltip (#26043)

Add conf parameter to CLI for airflow dags test (#25900)

Include scheduled slots in pools view (#26006)

Add output property to MappedOperator (#25604)

Add roles delete command to cli (#25854)

Add Airflow specific warning classes (#25799)

Add support for TaskGroup in ExternalTaskSensor (#24902)

Add @task.kubernetes taskflow decorator (#25663)

Add a way to import Airflow without side-effects (#25832)

Let timetables control generated run_ids. (#25795)

Allow per-timetable ordering override in grid view (#25633)

Grid logs for mapped instances (#25610, #25621, #25611)

Consolidate to one schedule param (#25410)

DAG regex flag in backfill command (#23870)

Adding support for owner links in the Dags view UI (#25280)

Ability to clear a specific DAG Run's task instances via REST API (#23516)

Possibility to document DAG with a separate markdown file (#25509)

Add parsing context to DAG Parsing (#25161)

Implement CronTriggerTimetable (#23662)

Add option to mask sensitive data in UI configuration page (#25346)

Create new databases from the ORM (#24156)

Implement XComArg.zip(*xcom_args) (#25176)

Introduce sla_miss metric (#23402)

Implement map() semantic (#25085)

Add override method to TaskGroupDecorator (#25160)

Implement expand_kwargs() (#24989)

Add parameter to turn off SQL query logging (#24570)

Add DagWarning model, and a check for missing pools (#23317)

Add Task Logs to Grid details panel (#24249)

Added small health check server and endpoint in scheduler(#23905)

Add built-in External Link for ExternalTaskMarker operator (#23964)

Add default task retry delay config (#23861)

Add clear DagRun endpoint. (#23451)

Add support for timezone as string in cron interval timetable (#23279)

Add auto-refresh to dags home page (#22900, #24770)

Improvements

Add more weekday operator and sensor examples #26071 (#26098)

Add subdir parameter to dags reserialize command (#26170)

Update zombie message to be more descriptive (#26141)

Only send an SlaCallbackRequest if the DAG is scheduled (#26089)

Promote Operator.output more (#25617)

Upgrade API files to typescript (#25098)

Less hacky double-rendering prevention in mapped task (#25924)

Improve Audit log (#25856)

Remove mapped operator validation code (#25870)

More DAG(schedule=...) improvements (#25648)

Reduce operator_name dupe in serialized JSON (#25819)

Make grid view group/mapped summary UI more consistent (#25723)

Remove useless statement in task_group_to_grid (#25654)

Add optional data interval to CronTriggerTimetable (#25503)

Remove unused code in /grid endpoint (#25481)

Add and document description fields (#25370)

Improve Airflow logging for operator Jinja template processing (#25452)

Update core example DAGs to use @task.branch decorator (#25242)

Update DAG audit_log route (#25415)

Change stdout and stderr access mode to append in commands (#25253)

Remove getTasks from Grid view (#25359)

Improve taskflow type hints with ParamSpec (#25173)

Use tables in grid details panes (#25258)

Explicitly list @dag arguments (#25044)

More typing in SchedulerJob and TaskInstance (#24912)

Patch getfqdn with more resilient version (#24981)

Replace all NBSP characters by whitespaces (#24797)

Re-serialize all DAGs on airflow db upgrade (#24518)

Rework contract of try_adopt_task_instances method (#23188)

Make expand() error vague so it's not misleading (#24018)

Add enum validation for [webserver]analytics_tool (#24032)

Add dttm searchable field in audit log (#23794)

Allow more parameters to be piped through via execute_in_subprocess (#23286)

Use func.count to count rows (#23657)

Remove stale serialized dags (#22917)

AIP45 Remove dag parsing in airflow run local (#21877)

Add support for queued state in DagRun update endpoint. (#23481)

Add fields to dagrun endpoint (#23440)

Use sql_alchemy_conn for celery result backend when result_backend is not set (#24496)

Bug Fixes

Have consistent types between the ORM and the migration files (#24044, #25869)

Disallow any dag tags longer than 100 char (#25196)

Add the dag_id to AirflowDagCycleException message (#26204)

Properly build URL to retrieve logs independently from system (#26337)

For worker log servers only bind to IPV6 when dual stack is available (#26222)

Fix TaskInstance.task not defined before handle_failure (#26040)

Undo secrets backend config caching (#26223)

Fix faulty executor config serialization logic (#26191)

Show DAGs and Datasets menu links based on role permission (#26183)

Allow setting TaskGroup tooltip via function docstring (#26028)

Fix RecursionError on graph view of a DAG with many tasks (#26175)

Fix backfill occasional deadlocking (#26161)

Fix DagRun.start_date not set during backfill with --reset-dagruns True (#26135)

Use label instead of id for dynamic task labels in graph (#26108)

Don't fail DagRun when leaf mapped_task is SKIPPED (#25995)

Add group prefix to decorated mapped task (#26081)

Fix UI flash when triggering with dup logical date (#26094)

Fix Make items nullable for TaskInstance related endpoints to avoid API errors (#26076)

Fix BranchDateTimeOperator to be timezone-awreness-insensitive (#25944)

Fix legacy timetable schedule interval params (#25999)

Fix response schema for list-mapped-task-instance (#25965)

Properly check the existence of missing mapped TIs (#25788)

Fix broken auto-refresh on grid view (#25950)

Use per-timetable ordering in grid UI (#25880)

Rewrite recursion when parsing DAG into iteration (#25898)

Find cross-group tasks in iter_mapped_dependants (#25793)

Fail task if mapping upstream fails (#25757)

Support / in variable get endpoint (#25774)

Use cfg default_wrap value for grid logs (#25731)

Add origin request args when triggering a run (#25729)

Operator name separate from class (#22834)

Fix incorrect data interval alignment due to assumption on input time alignment (#22658)

Return None if an XComArg fails to resolve (#25661)

Correct json arg help in airflow variables set command (#25726)

Added MySQL index hint to use ti_state on find_zombies query (#25725)

Only excluded actually expanded fields from render (#25599)

Grid, fix toast for axios errors (#25703)

Fix UI redirect (#26409)

Require dag_id arg for dags list-runs (#26357)

Check for queued states for dags auto-refresh (#25695)

Fix upgrade code for the dag_owner_attributes table (#25579)

Add map index to task logs api (#25568)

Ensure that zombie tasks for dags with errors get cleaned up (#25550)

Make extra link work in UI (#25500)

Sync up plugin API schema and definition (#25524)

First/last names can be empty (#25476)

Refactor DAG pages to be consistent (#25402)

Check expand_kwargs() input type before unmapping (#25355)

Filter XCOM by key when calculating map lengths (#24530)

Fix ExternalTaskSensor not working with dynamic task (#25215)

Added exception catching to send default email if template file raises any exception (#24943)

Bring MappedOperator members in sync with BaseOperator (#24034)

Misc/Internal

Add automatically generated ERD schema for the MetaData DB (#26217)

Mark serialization functions as internal (#26193)

Remove remaining deprecated classes and replace them with PEP562 (#26167)

Move dag_edges and task_group_to_dict to corresponding util modules (#26212)

Lazily import many modules to improve import speed (#24486, #26239)

FIX Incorrect typing information (#26077)

Add missing contrib classes to deprecated dictionaries (#26179)

Re-configure/connect the ORM after forking to run a DAG processor (#26216)

Remove cattrs from lineage processing. (#26134)

Removed deprecated contrib files and replace them with PEP-562 getattr (#26153)

Make BaseSerialization.serialize "public" to other classes. (#26142)

Change the template to use human readable task_instance description (#25960)

Bump moment-timezone from 0.5.34 to 0.5.35 in /airflow/www (#26080)

Fix Flask deprecation warning (#25753)

Add CamelCase to generated operations types (#25887)

Fix migration issues and tighten the CI upgrade/downgrade test (#25869)

Fix type annotations in SkipMixin (#25864)

Workaround setuptools editable packages path issue (#25848)

Bump undici from 5.8.0 to 5.9.1 in /airflow/www (#25801)

Add custom_operator_name attr to _BranchPythonDecoratedOperator (#25783)

Clarify filename_template deprecation message (#25749)

Use ParamSpec to replace ... in Callable (#25658)

Remove deprecated modules (#25543)

Documentation on task mapping additions (#24489)

Remove Smart Sensors (#25507)

Fix elasticsearch test config to avoid warning on deprecated template (#25520)

Bump terser from 4.8.0 to 4.8.1 in /airflow/ui (#25178)

Generate typescript types from rest API docs (#25123)

Upgrade utils files to typescript (#25089)

Upgrade remaining context file to typescript. (#25096)

Migrate files to ts (#25267)

Upgrade grid Table component to ts. (#25074)

Skip mapping against mapped ti if it returns None (#25047)

Refactor js file structure (#25003)

Move mapped kwargs introspection to separate type (#24971)

Only assert stuff for mypy when type checking (#24937)

Bump moment from 2.29.3 to 2.29.4 in /airflow/www (#24885)

Remove "bad characters" from our codebase (#24841)

Remove xcom_push flag from BashOperator (#24824)

Move Flask hook registration to end of file (#24776)

Upgrade more javascript files to typescript (#24715)

Clean up task decorator type hints and docstrings (#24667)

Preserve original order of providers' connection extra fields in UI (#24425)

Rename charts.css to chart.css (#24531)

Rename grid.css to chart.css (#24529)

Misc: create new process group by set_new_process_group utility (#24371)

Airflow UI fix Prototype Pollution (#24201)

Bump moto version (#24222)

Remove unused [github_enterprise] from ref docs (#24033)

Clean up f-strings in logging calls (#23597)

Add limit for JPype1 (#23847)

Simply json responses (#25518)

Add min attrs version (#26408)

Doc only changes

Add url prefix setting for Celery Flower (#25986)

Updating deprecated configuration in examples (#26037)

Fix wrong link for taskflow tutorial (#26007)

Reorganize tutorials into a section (#25890)

Fix concept doc for dynamic task map (#26002)

Update code examples from "classic" operators to taskflow (#25845, #25657)

Add instructions on manually fixing MySQL Charset problems (#25938)

Prefer the local Quick Start in docs (#25888)

Fix broken link to Trigger Rules (#25840)

Improve docker documentation (#25735)

Correctly link to Dag parsing context in docs (#25722)

Add note on task_instance_mutation_hook usage (#25607)

Note that TaskFlow API automatically passes data between tasks (#25577)

Update DAG run to clarify when a DAG actually runs (#25290)

Update tutorial docs to include a definition of operators (#25012)

Rewrite the Airflow documentation home page (#24795)

Fix task-generated mapping example (#23424)

Add note on subtle logical date change in 2.2.0 (#24413)

Add missing import in best-practices code example (#25391)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.4.0-source.tar.gz(21.95 MB)
apache-airflow-2.4.0-source.tar.gz.asc(870 bytes)
apache-airflow-2.4.0-source.tar.gz.sha512(165 bytes)
apache-airflow-2.4.0.tar.gz(5.68 MB)
apache-airflow-2.4.0.tar.gz.asc(870 bytes)
apache-airflow-2.4.0.tar.gz.sha512(158 bytes)
apache_airflow-2.4.0-py3-none-any.whl(6.19 MB)
apache_airflow-2.4.0-py3-none-any.whl.asc(870 bytes)
apache_airflow-2.4.0-py3-none-any.whl.sha512(168 bytes)
2.3.4(Aug 23, 2022)
Bug Fixes

Disable attrs state management on MappedOperator (#24772)

Serialize pod_override to JSON before pickling executor_config (#24356)

Fix pid check (#24636)

Rotate session id during login (#25771)

Fix mapped sensor with reschedule mode (#25594)

Cache the custom secrets backend so the same instance gets re-used (#25556)

Add right padding (#25554)

Fix reducing mapped length of a mapped task at runtime after a clear (#25531)

Fix airflow db reset when dangling tables exist (#25441)

Change disable_verify_ssl behaviour (#25023)

Set default task group in dag.add_task method (#25000)

Removed interfering force of index. (#25404)

Remove useless logging line (#25347)

Adding mysql index hint to use index on task_instance.state in critical section query (#25673)

Configurable umask to all daemonized processes. (#25664)

Fix the errors raised when None is passed to template filters (#25593)

Allow wildcarded CORS origins (#25553)

Fix "This Session's transaction has been rolled back" (#25532)

Fix Serialization error in TaskCallbackRequest (#25471)

fix - resolve bash by absolute path (#25331)

Add __repr__ to ParamsDict class (#25305)

Only load distribution of a name once (#25296)

convert TimeSensorAsync target_time to utc on call time (#25221)

call updateNodeLabels after expandGroup (#25217)

Stop SLA callbacks gazumping other callbacks and DOS'ing the DagProcessorManager queue (#25147)

Fix invalidateQueries call (#25097)

airflow/www/package.json: Add name, version fields. (#25065)

No grid auto-refresh for backfill dag runs (#25042)

Fix tag link on dag detail page (#24918)

Fix zombie task handling with multiple schedulers (#24906)

Bind log server on worker to IPv6 address (#24755) (#24846)

Add %z for %(asctime)s to fix timezone for logs on UI (#24811)

TriggerDagRunOperator.operator_extra_links is attr (#24676)

Send DAG timeout callbacks to processor outside of prohibit_commit (#24366)

Don't rely on current ORM structure for db clean command (#23574)

Clear next method when clearing TIs (#23929)

Two typing fixes (#25690)

Doc only changes

Update set-up-database.rst (#24983)

Fix syntax in mysql setup documentation (#24893 (#24939)

Note how DAG policy works with default_args (#24804)

Update PythonVirtualenvOperator Howto (#24782)

Doc: Add hyperlinks to Github PRs for Release Notes (#24532)

Misc/Internal

Remove depreciation warning when use default remote tasks logging handlers (#25764)

clearer method name in scheduler_job.py (#23702)

Bump cattrs version (#25689)

Include missing mention of external_executor_id in sql_engine_collation_for_ids docs (#25197)

Refactor DR.task_instance_scheduling_decisions (#24774)

Sort operator extra links (#24992)

Extends resolve_xcom_backend function level documentation (#24965)

Upgrade FAB to 4.1.3 (#24884)

Limit Flask to <2.3 in the wake of 2.2 breaking our tests (#25511)

Limit astroid version to < 2.12 (#24982)

Move javascript compilation to host (#25169)

Bump typing-extensions and mypy for ParamSpec (#25088)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.3.4-source.tar.gz(20.31 MB)
apache-airflow-2.3.4-source.tar.gz.asc(699 bytes)
apache-airflow-2.3.4-source.tar.gz.sha512(165 bytes)
apache-airflow-2.3.4.tar.gz(4.53 MB)
apache-airflow-2.3.4.tar.gz.asc(699 bytes)
apache-airflow-2.3.4.tar.gz.sha512(158 bytes)
apache_airflow-2.3.4-py3-none-any.whl(5.20 MB)
apache_airflow-2.3.4-py3-none-any.whl.asc(699 bytes)
apache_airflow-2.3.4-py3-none-any.whl.sha512(168 bytes)
2.3.3(Jul 9, 2022)
Bug Fixes

Fix exception in mini task scheduler (#24865)

Fix cycle bug with attaching label to task group (#24847)

Fix timestamp defaults for sensorinstance (#24638)

Move fallible ti.task.dag assignment back inside try/except block (#24533) (#24592)

Add missing types to FSHook (#24470)

Mask secrets in stdout for airflow tasks test (#24362)

DebugExecutor use ti.run() instead of ti._run_raw_task (#24357)

Fix bugs in URI constructor for MySQL connection (#24320)

Missing scheduleinterval nullable true added in openapi (#24253)

Unify return_code interface for task runner (#24093)

Handle occasional deadlocks in trigger with retries (#24071)

Remove special serde logic for mapped op_kwargs (#23860)

ExternalTaskSensor respects soft_fail if the external task enters a failed_state (#23647)

Fix StatD timing metric units (#21106)

Add cache_ok flag to sqlalchemy TypeDecorators. (#24499)

Allow for LOGGING_LEVEL=DEBUG (#23360)

Fix grid date ticks (#24738, #24849)

Debounce status highlighting in Grid view (#24710)

Fix Grid vertical scrolling (#24684)

don't try to render child rows for closed groups (#24637)

Do not calculate grid root instances (#24528)

Maintain grid view selection on filtering upstream (#23779)

Speed up grid_data endpoint by 10x (#24284)

Apply per-run log templates to log handlers (#24153)

Don't crash scheduler if exec config has old k8s objects (#24117)

TI.log_url fix for map_index (#24335)

Fix migration 0080_2_0_2 - Replace null values before setting column not null (#24585)

Patch sql_alchemy_conn if old Postgres schemes used (#24569)

Seed log_template table (#24511)

Fix deprecated log_id_template value (#24506)

Fix toast messages (#24505)

Add indexes for CASCADE deletes for task_instance (#24488)

Return empty dict if Pod JSON encoding fails (#24478)

Improve grid rendering performance with a custom tooltip (#24417, #24449)

Check for run_id for grid group summaries (#24327)

Optimize calendar view for cron scheduled DAGs (#24262)

Use get_hostname instead of socket.getfqdn (#24260)

Check that edge nodes actually exist (#24166)

Fix useTasks crash on error (#24152)

Do not fail re-queued TIs (#23846)

Reduce grid view API calls (#24083)

Rename Permissions to Permission Pairs. (#24065)

Replace use_task_execution_date with use_task_logical_date (#23983)

Grid fix details button truncated and small UI tweaks (#23934)

Add TaskInstance State REMOVED to finished states and success states (#23797)

Fix mapped task immutability after clear (#23667)

Fix permission issue for dag that has dot in name (#23510)

Fix closing connection dbapi.get_pandas_df (#23452)

Check bag DAG schedule_interval match timetable (#23113)

Parse error for task added to multiple groups (#23071)

Fix flaky order of returned dag runs (#24405)

Migrate jsx files that affect run/task selection to tsx (#24509)

Fix links to sources for examples (#24386)

Set proper Content-Type and chartset on grid_data endpoint (#24375)

Doc only changes

Fix doc description of [core] parallelism config setting (#23768)

Update templates doc to mention extras and format Airflow Vars / Conns (#24735)

Document built in Timetables (#23099)

Alphabetizes two tables (#23923)

Clarify that users should not use Maria DB (#24556)

Add imports to deferring code samples (#24544)

Add note about image regeneration in June 2022 (#24524)

Small cleanup of get_current_context() chapter (#24482)

Fix default 2.2.5 log_id_template (#24455)

Update description of installing providers separately from core (#24454)

Mention context variables and logging (#24304)

Misc/Internal

Do not support MSSQL less than v2017 in code (#24095)

Drop Python 3.6 compatibility objects/modules (#24048)

Remove upper-binding for SQLAlchemy (#24819)

Remove internet explorer support (#24495)

Removing magic status code numbers from api_connexion (#24050)

Upgrade FAB to 4.1.2 (#24619)

Switch Markdown engine to markdown-it-py (#19702)

Update rich to latest version across the board. (#24186)

Get rid of TimedJSONWebSignatureSerializer (#24519)

Update flask-appbuilder authlib/ oauth dependency (#24516)

Upgrade to webpack 5 (#24485)

Add typescript (#24337)

The JWT claims in the request to retrieve logs have been standardized: we use nbf and aud claims for maturity and audience of the requests. Also "filename" payload field is used to keep log name. (#24519)

Address all yarn test warnings (#24722)

Upgrade to react 18 and chakra 2 (#24430)

Refactor DagRun.verify_integrity (#24114)

Upgrade FAB to 4.1.1 (#24399)

We now need at least Flask-WTF 0.15 (#24621)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.3.3-source.tar.gz(20.31 MB)
apache-airflow-2.3.3-source.tar.gz.asc(699 bytes)
apache-airflow-2.3.3-source.tar.gz.sha512(165 bytes)
apache-airflow-2.3.3.tar.gz(4.52 MB)
apache-airflow-2.3.3.tar.gz.asc(699 bytes)
apache-airflow-2.3.3.tar.gz.sha512(158 bytes)
apache_airflow-2.3.3-py3-none-any.whl(5.20 MB)
apache_airflow-2.3.3-py3-none-any.whl.asc(699 bytes)
apache_airflow-2.3.3-py3-none-any.whl.sha512(168 bytes)
2.3.2(Jun 4, 2022)
Bug Fixes

Run the check_migration loop at least once

Fix grid view for mapped tasks (#24059)

Icons in grid view for different DAG run types (#23970)

Faster grid view (#23951)

Disallow calling expand with no arguments (#23463)

Add missing is_mapped field to Task response. (#23319)

DagFileProcessorManager: Start a new process group only if current process not a session leader (#23872)

Mask sensitive values for not-yet-running TIs (#23807)

Add cascade to dag_tag to dag foreign key (#23444)

Use --subdir argument value for standalone dag processor. (#23864)

Highlight task states by hovering on legend row (#23678)

Fix and speed up grid view (#23947)

Prevent UI from crashing if grid task instances are null (#23939)

Remove redundant register exit signals in dag-processor command (#23886)

Add __wrapped__ property to _TaskDecorator (#23830)

Fix UnboundLocalError when sql is empty list in DbApiHook (#23816)

Enable clicking on DAG owner in autocomplete dropdown (#23804)

Simplify flash message for _airflow_moved tables (#23635)

Exclude missing tasks from the gantt view (#23627)

Doc only changes

Add column names for DB Migration Reference (#23853)

Misc/Internal

Remove pinning for xmltodict (#23992)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.3.2-source.tar.gz(20.55 MB)
apache-airflow-2.3.2-source.tar.gz.asc(699 bytes)
apache-airflow-2.3.2-source.tar.gz.sha512(165 bytes)
apache-airflow-2.3.2.tar.gz(4.29 MB)
apache-airflow-2.3.2.tar.gz.asc(699 bytes)
apache-airflow-2.3.2.tar.gz.sha512(158 bytes)
apache_airflow-2.3.2-py3-none-any.whl(4.97 MB)
apache_airflow-2.3.2-py3-none-any.whl.asc(699 bytes)
apache_airflow-2.3.2-py3-none-any.whl.sha512(168 bytes)
2.3.1(May 25, 2022)
Bug Fixes

Automatically reschedule stalled queued tasks in CeleryExecutor (#23690)

Fix expand/collapse all buttons (#23590)

Grid view status filters (#23392)

Expand/collapse all groups (#23487)

Fix retrieval of deprecated non-config values (#23723)

Fix secrets rendered in UI when task is not executed. (#22754)

Fix provider import error matching (#23825)

Fix regression in ignoring symlinks (#23535)

Fix dag-processor fetch metadata database config (#23575)

Fix auto upstream dep when expanding non-templated field (#23771)

Fix task log is not captured (#23684)

Add reschedule to the serialized fields for the BaseSensorOperator (#23674)

Modify db clean to also catch the ProgrammingError exception (#23699)

Remove titles from link buttons (#23736)

Fix grid details header text overlap (#23728)

Ensure execution_timeout as timedelta (#23655)

Don't run pre-migration checks for downgrade (#23634)

Add index for event column in log table (#23625)

Implement send_callback method for CeleryKubernetesExecutor and LocalKubernetesExecutor (#23617)

Fix PythonVirtualenvOperator templated_fields (#23559)

Apply specific ID collation to root_dag_id too (#23536)

Prevent KubernetesJobWatcher getting stuck on resource too old (#23521)

Fix scheduler crash when expanding with mapped task that returned none (#23486)

Fix broken dagrun links when many runs start at the same time (#23462)

Fix: Exception when parsing log #20966 (#23301)

Handle invalid date parsing in webserver views. (#23161)

Pools with negative open slots should not block other pools (#23143)

Move around overflow, position and padding (#23044)

Change approach to finding bad rows to LEFT OUTER JOIN. (#23528)

Only count bad refs when moved table exists (#23491)

Visually distinguish task group summary (#23488)

Remove color change for highly nested groups (#23482)

Optimize 2.3.0 pre-upgrade check queries (#23458)

Add backward compatibility for core__sql_alchemy_conn__cmd (#23441)

Fix literal cross product expansion (#23434)

Fix broken task instance link in xcom list (#23367)

Fix connection test button (#23345)

fix cli airflow dags show for mapped operator (#23339)

Hide some task instance attributes (#23338)

Don't show grid actions if server would reject with permission denied (#23332)

Use run_id for ti.mark_success_url (#23330)

Fix update user auth stats (#23314)

Use <Time /> in Mapped Instance table (#23313)

Fix duplicated Kubernetes DeprecationWarnings (#23302)

Store grid view selection in url params (#23290)

Remove custom signal handling in Triggerer (#23274)

Override pool for TaskInstance when pool is passed from cli. (#23258)

Show warning if '/' is used in a DAG run ID (#23106)

Use kubernetes queue in kubernetes hybrid executors (#23048)

Add tags inside try block. (#21784)

Doc only changes

Move dag_processing.processor_timeouts to counters section (#23393)

Clarify that bundle extras should not be used for PyPi installs (#23697)

Synchronize support for Postgres and K8S in docs (#23673)

Replace DummyOperator references in docs (#23502)

Add doc notes for keyword-only args for expand() and partial() (#23373)

Document fix for broken elasticsearch logs with 2.3.0+ upgrade (#23821)

Misc/Internal

Add typing for airflow/configuration.py (#23716)

Disable Flower by default from docker-compose (#23685)

Added postgres 14 to support versions(including breeze) (#23506)

add K8S 1.24 support (#23637)

Refactor code references from tree to grid (#23254)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.3.1-source.tar.gz(20.55 MB)
apache-airflow-2.3.1-source.tar.gz.asc(699 bytes)
apache-airflow-2.3.1-source.tar.gz.sha512(165 bytes)
apache-airflow-2.3.1.tar.gz(4.29 MB)
apache-airflow-2.3.1.tar.gz.asc(699 bytes)
apache-airflow-2.3.1.tar.gz.sha512(158 bytes)
apache_airflow-2.3.1-py3-none-any.whl(4.97 MB)
apache_airflow-2.3.1-py3-none-any.whl.asc(699 bytes)
apache_airflow-2.3.1-py3-none-any.whl.sha512(168 bytes)
helm-chart/1.6.0(May 20, 2022)
New Features

Support annotations on volumeClaimTemplates (#23433)

Add support for topologySpreadConstraints to Helm Chart (#22712)

Helm support for LocalKubernetesExecutor (#22388)

Add securityContext config for Redis to Helm chart (#22182)

Allow annotations on Helm DAG PVC (#22261)

enable optional subPath for DAGs volume mount (#22323)

Added support to override auth_type in auth_file in PgBouncer Helm configuration (#21999)

Add extraVolumeMounts to Flower (#22414)

Add webserver PodDisruptionBudget (#21735)

Improvements

Ensure the messages from migration job show up early (#23479)

Allow migration jobs and init containers to be optional (#22195)

Use jobs check command for liveness probe check in Airflow 2 (#22143)

Doc only changes

Adds resultBackendSecretName warning in Helm production docs (#23307)

Misc

Update default Airflow version to 2.3.0 (#23386)

Move the database configuration to a new section (#22284)

Disable flower in chart by default (#23737)

Source code(tar.gz)
Source code(zip)
airflow-1.6.0.tgz(146.52 KB)
airflow-1.6.0.tgz.asc(870 bytes)
airflow-1.6.0.tgz.prov(5.69 KB)
airflow-1.6.0.tgz.sha512(148 bytes)
airflow-chart-1.6.0-source.tar.gz(98.26 KB)
airflow-chart-1.6.0-source.tar.gz.asc(870 bytes)
airflow-chart-1.6.0-source.tar.gz.sha512(164 bytes)
2.3.0(May 1, 2022)
New Features

Add dynamic task mapping (https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-42+milestone%3A%22Airflow+2.3.0%22)

New Grid View replaces Tree View (#18675)

Templated requirements.txt in Python Operators (#17349)

Allow reuse of decorated tasks (#22941)

Move the database configuration to a new section (#22284)

Add SmoothOperator (#22813)

Make operator's execution_timeout configurable (#22389)

Events Timetable (#22332)

Support dag serialization with custom ti_deps rules (#22698)

Support log download in task log view (#22804)

support for continue backfill on failures (#22697)

Add dag-processor cli command (#22305)

Add possibility to create users in LDAP mode (#22619)

Add ignore_first_depends_on_past for scheduled jobs (#22491)

Update base sensor operator to support XCOM return value (#20656)

Add an option for run id in the ui trigger screen (#21851)

Enable JSON serialization for connections (#19857)

Add REST API endpoint for bulk update of DAGs (#19758)

Add queue button to click-on-DagRun interface. (#21555)

Add list-import-errors to airflow dags command (#22084)

Store callbacks in database if standalone_dag_processor config is True. (#21731)

Add LocalKubernetesExecutor (#19729)

Add celery.task_timeout_error metric (#21602)

Airflow db downgrade cli command (#21596)

Add ALL_SKIPPED trigger rule (#21662)

Add db clean CLI command for purging old data (#20838)

Add celery_logging_level (#21506)

Support different timeout value for dag file parsing (#21501)

Support generating SQL script for upgrades (#20962)

Add option to compress Serialized dag data (#21332)

Branch python operator decorator (#20860)

Add Audit Log View to Dag View (#20733)

Add missing StatsD metric for failing SLA Callback notification (#20924)

Add ShortCircuitOperator configurability for respecting downstream trigger rules (#20044)

Allow using Markup in page title in Webserver (#20888)

Add Listener Plugin API that tracks TaskInstance state changes (#20443)

Add context var hook to inject more env vars (#20361)

Add a button to set all tasks to skipped (#20455)

Cleanup pending pods (#20438)

Add config to warn public deployment exposure in UI (#18557)

Log filename template records (#20165)

Added windows extensions (#16110)

Showing approximate time until next dag_run in Airflow (#20273)

Extend config window on UI (#20052)

Add show dag dependencies feature to CLI (#19985)

Add cli command for 'airflow dags reserialize` (#19471)

Add missing description field to Pool schema(REST API) (#19841)

Introduce DagRun action to change state to queued. (#19353)

Add DAG run details page (#19705)

Add role export/import to cli tools (#18916)

Adding dag_id_pattern parameter to the /dags endpoint (#18924)

Improvements

Show schedule_interval/timetable description in UI (#16931)

Added column duration to DAG runs view (#19482)

Enable use of custom conn extra fields without prefix (#22607)

Initialize finished counter at zero (#23080)

Improve logging of optional provider features messages (#23037)

Meaningful error message in resolve_template_files (#23027)

Update ImportError items instead of deleting and recreating them (#22928)

Add option --skip-init to db reset command (#22989)

Support importing connections from files with ".yml" extension (#22872)

Support glob syntax in .airflowignore files (#21392) (#22051)

Hide pagination when data is a single page (#22963)

Support for sorting DAGs in the web UI (#22671)

Speed up has_access decorator by ~200ms (#22858)

Add XComArg to lazy-imported list of Airflow module (#22862)

Add more fields to REST API dags/dag_id/details endpoint (#22756)

Don't show irrelevant/duplicated/"internal" Task attrs in UI (#22812)

No need to load whole ti in current_state (#22764)

Pickle dag exception string fix (#22760)

Better verification of Localexecutor's parallelism option (#22711)

log backfill exceptions to sentry (#22704)

retry commit on MySQL deadlocks during backfill (#22696)

Add more fields to REST API get DAG(dags/dag_id) endpoint (#22637)

Use timetable to generate planned days for current year (#22055)

Disable connection pool for celery worker (#22493)

Make date picker label visible in trigger dag view (#22379)

Expose try_number in airflow vars (#22297)

Add generic connection type (#22310)

Add a few more fields to the taskinstance finished log message (#22262)

Pause auto-refresh if scheduler isn't running (#22151)

Show DagModel details. (#21868)

Add pip_install_options to PythonVirtualenvOperator (#22158)

Show import error for airflow dags list CLI command (#21991)

Pause auto-refresh when page is hidden (#21904)

Default args type check (#21809)

Enhance magic methods on XComArg for UX (#21882)

py files don't have to be checked is_zipfiles in refresh_dag (#21926)

Fix TaskDecorator type hints (#21881)

Add 'Show record' option for variables (#21342)

Use DB where possible for quicker airflow dag subcommands (#21793)

REST API: add rendered fields in task instance. (#21741)

Change the default auth backend to session (#21640)

Don't check if py DAG files are zipped during parsing (#21538)

Switch XCom implementation to use run_id (#20975)

Action log on Browse Views (#21569)

Implement multiple API auth backends (#21472)

Change logging level details of connection info in get_connection() (#21162)

Support mssql in airflow db shell (#21511)

Support config worker_enable_remote_control for celery (#21507)

Log memory usage in CgroupTaskRunner (#21481)

Modernize DAG-related URL routes and rename "tree" to "grid" (#20730)

Move Zombie detection to SchedulerJob (#21181)

Improve speed to run airflow by 6x (#21438)

Add more SQL template fields renderers (#21237)

Simplify fab has access lookup (#19294)

Log context only for default method (#21244)

Log trigger status only if at least one is running (#21191)

Add optional features in providers. (#21074)

Better multiple_outputs inferral for @task.python (#20800)

Improve handling of string type and non-attribute template_fields (#21054)

Remove un-needed deps/version requirements (#20979)

Correctly specify overloads for TaskFlow API for type-hinting (#20933)

Introduce notification_sent to SlaMiss view (#20923)

Rewrite the task decorator as a composition (#20868)

Add "Greater/Smaller than or Equal" to filters in the browse views (#20602) (#20798)

Rewrite DAG run retrieval in task command (#20737)

Speed up creation of DagRun for large DAGs (5k+ tasks) by 25-130% (#20722)

Make native environment Airflow-flavored like sandbox (#20704)

Better error when param value has unexpected type (#20648)

Add filter by state in DagRun REST API (List Dag Runs) (#20485)

Prevent exponential memory growth in Tasks with custom logging handler (#20541)

Set default logger in logging Mixin (#20355)

Reduce deprecation warnings from www (#20378)

Add hour and minute to time format on x-axis of all charts using nvd3.lineChart (#20002)

Add specific warning when Task asks for more slots than pool defined with (#20178)

UI: Update duration column for better human readability (#20112)

Use Viewer role as example public role (#19215)

Properly implement DAG param dict copying (#20216)

ShortCircuitOperator push XCom by returning python_callable result (#20071)

Add clear logging to tasks killed due to a Dagrun timeout (#19950)

Change log level for Zombie detection messages (#20204)

Better confirmation prompts (#20183)

Only execute TIs of running DagRuns (#20182)

Check and run migration in commands if necessary (#18439)

Log only when Zombies exists (#20118)

Increase length of the email and username (#19932)

Add more filtering options for TI's in the UI (#19910)

Dynamically enable "Test Connection" button by connection type (#19792)

Avoid littering postgres server logs with "could not obtain lock" with HA schedulers (#19842)

Renamed Connection.get_hook parameter to make it the same as in SqlSensor and SqlOperator. (#19849)

Add hook_params in SqlSensor using the latest changes from PR #18718. (#18431)

Speed up webserver boot time by delaying provider initialization (#19709)

Configurable logging of XCOM value in PythonOperator (#19378)

Minimize production js files (#19658)

Add hook_params in BaseSqlOperator (#18718)

Add missing "end_date" to hash components (#19281)

More friendly output of the airflow plugins command + add timetables (#19298)

Add sensor default timeout config (#19119)

Update taskinstance REST API schema to include dag_run_id field (#19105)

Adding feature in bash operator to append the user defined env variable to system env variable (#18944)

Duplicate Connection: Added logic to query if a connection id exists before creating one (#18161)

Bug Fixes

Use inherited 'trigger_tasks' method (#23016)

In DAG dependency detector, use class type instead of class name (#21706)

Fix tasks being wrongly skipped by schedule_after_task_execution (#23181)

Fix X-Frame enabled behaviour (#23222)

Allow extra to be nullable in connection payload as per schema(REST API). (#23183)

Fix dag_id extraction for dag level access checks in web ui (#23015)

Fix timezone display for logs on UI (#23075)

Include message in graph errors (#23021)

Change trigger dropdown left position (#23013)

Don't add planned tasks for legacy DAG runs (#23007)

Add dangling rows check for TaskInstance references (#22924)

Validate the input params in connection CLI command (#22688)

Fix trigger event payload is not persisted in db (#22944)

Drop "airflow moved" tables in command db reset (#22990)

Add max width to task group tooltips (#22978)

Add template support for external_task_ids. (#22809)

Allow DagParam to hold falsy values (#22964)

Fix regression in pool metrics (#22939)

Priority order tasks even when using pools (#22483)

Do not clear XCom when resuming from deferral (#22932)

Handle invalid JSON metadata in get_logs_with_metadata endpoint. (#22898)

Fix pre-upgrade check for rows dangling w.r.t. dag_run (#22850)

Fixed backfill interference with scheduler (#22701)

Support conf param override for backfill runs (#22837)

Correctly interpolate pool name in PoolSlotsAvailableDep statues (#22807)

Fix email_on_failure with render_template_as_native_obj (#22770)

Fix processor cleanup on DagFileProcessorManager (#22685)

Prevent meta name clash for task instances (#22783)

remove json parse for gantt chart (#22780)

Check for missing dagrun should know version (#22752)

Fixes ScheduleInterval spec (#22635)

Fixing task status for non-running and non-committed tasks (#22410)

Do not log the hook connection details even at DEBUG level (#22627)

Stop crashing when empty logs are received from kubernetes client (#22566)

Fix bugs about timezone change (#22525)

Fix entire DAG stops when one task has end_date (#20920)

Use logger to print message during task execution. (#22488)

Make sure finalizers are not skipped during exception handling (#22475)

update smart sensor docs and minor fix on is_smart_sensor_compatible() (#22386)

Fix run_id k8s and elasticsearch compatibility with Airflow 2.1 (#22385)

Allow to except_skip None on BranchPythonOperator (#20411)

Fix incorrect datetime details (DagRun views) (#21357)

Remove incorrect deprecation warning in secrets backend (#22326)

Remove RefreshConfiguration workaround for K8s token refreshing (#20759)

Masking extras in GET /connections/<connection> endpoint (#22227)

Set queued_dttm when submitting task to directly to executor (#22259)

Addressed some issues in the tutorial mentioned in discussion #22233 (#22236)

Change default python executable to python3 for docker decorator (#21973)

Don't validate that Params are JSON when NOTSET (#22000)

Add per-DAG delete permissions (#21938)

Fix handling some None parameters in kubernetes 23 libs. (#21905)

Fix handling of empty (None) tags in bulk_write_to_db (#21757)

Fix DAG date range bug (#20507)

Removed request.referrer from views.py (#21751)

Make DbApiHook use get_uri from Connection (#21764)

Fix some migrations (#21670)

[de]serialize resources on task correctly (#21445)

Add params dag_id, task_id etc to XCom.serialize_value (#19505)

Update test connection functionality to use custom form fields (#21330)

fix all "high" npm vulnerabilities (#21526)

Fix bug incorrectly removing action from role, rather than permission. (#21483)

Fix relationship join bug in FAB/SecurityManager with SQLA 1.4 (#21296)

Use Identity instead of Sequence in SQLAlchemy 1.4 for MSSQL (#21238)

Ensure on_task_instance_running listener can get at task (#21157)

Return to the same place when triggering a DAG (#20955)

Fix task ID deduplication in @task_group (#20870)

Add downgrade to some FAB migrations (#20874)

Only validate Params when DAG is triggered (#20802)

Fix airflow trigger cli (#20781)

Fix task instances iteration in a pool to prevent blocking (#20816)

Allow depending to a @task_group as a whole (#20671)

Use original task's start_date if a task continues after deferral (#20062)

Disabled edit button in task instances list view page (#20659)

Fix a package name import error (#20519) (#20519)

Remove execution_date label when get cleanup pods list (#20417)

Remove unneeded FAB REST API endpoints (#20487)

Fix parsing of Cloudwatch log group arn containing slashes (#14667) (#19700)

Sanity check for MySQL's TIMESTAMP column (#19821)

Allow using default celery command group with executors subclassed from Celery-based executors. (#18189)

Move class_permission_name to mixin so it applies to all classes (#18749)

Adjust trimmed_pod_id and replace '.' with '-' (#19036)

Pass custom_headers to send_email and send_email_smtp (#19009)

Ensure catchup=False is used in example dags (#19396)

Edit permalinks in OpenApi description file (#19244)

Navigate directly to DAG when selecting from search typeahead list (#18991)

[Minor] Fix padding on home page (#19025)

Doc only changes

Update doc for DAG file processing (#23209)

Replace changelog/updating with release notes and towncrier now (#22003)

Fix wrong reference in tracking-user-activity.rst (#22745)

Remove references to rbac = True from docs (#22725)

Doc: Update description for executor-bound dependencies (#22601)

Update check-health.rst (#22372)

Stronger language about Docker Compose customizability (#22304)

Update logging-tasks.rst (#22116)

Add example config of sql_alchemy_connect_args (#22045)

Update best-practices.rst (#22053)

Add information on DAG pausing/deactivation/deletion (#22025)

Add brief examples of integration test dags you might want (#22009)

Run inclusive language check on CHANGELOG (#21980)

Add detailed email docs for Sendgrid (#21958)

Add docs for db upgrade / db downgrade (#21879)

Update modules_management.rst (#21889)

Fix UPDATING section on SqlAlchemy 1.4 scheme changes (#21887)

Update TaskFlow tutorial doc to show how to pass "operator-level" args. (#21446)

Fix doc - replace decreasing by increasing (#21805)

Add another way to dynamically generate DAGs to docs (#21297)

Add extra information about time synchronization needed (#21685)

Update debug.rst docs (#21246)

Replaces the usage of postgres:// with postgresql:// (#21205)

Fix task execution process in CeleryExecutor docs (#20783)

Misc/Internal

Bring back deprecated security manager functions (#23243)

Replace usage of DummyOperator with EmptyOperator (#22974)

Deprecate DummyOperator in favor of EmptyOperator (#22832)

Remove unnecessary python 3.6 conditionals (#20549)

Bump moment from 2.29.1 to 2.29.2 in /airflow/www (#22873)

Bump prismjs from 1.26.0 to 1.27.0 in /airflow/www (#22823)

Bump nanoid from 3.1.23 to 3.3.2 in /airflow/www (#22803)

Bump minimist from 1.2.5 to 1.2.6 in /airflow/www (#22798)

Remove dag parsing from db init command (#22531)

Update our approach for executor-bound dependencies (#22573)

Use Airflow.Base.metadata in FAB models (#22353)

Limit docutils to make our documentation pretty again (#22420)

Add Python 3.10 support (#22050)

[FEATURE] add 1.22 1.23 K8S support (#21902)

Remove pandas upper limit now that SQLA is 1.4+ (#22162)

Patch sql_alchemy_conn if old postgres scheme used (#22333)

Protect against accidental misuse of XCom.get_value() (#22244)

Order filenames for migrations (#22168)

Don't try to auto generate migrations for Celery tables (#22120)

Require SQLAlchemy 1.4 (#22114)

bump sphinx-jinja (#22101)

Add compat shim for SQLAlchemy to avoid warnings (#21959)

Rename xcom.dagrun_id to xcom.dag_run_id (#21806)

Deprecate non-JSON conn.extra (#21816)

Bump upper bound version of jsonschema to 5.0 (#21712)

Deprecate helper utility days_ago (#21653)

Remove :type lines now sphinx-autoapi supports type hints (#20951)

Silence deprecation warning in tests (#20900)

Use DagRun.run_id instead of execution_date when updating state of TIs (UI & REST API) (#18724)

Add Context stub to Airflow packages (#20817)

Update Kubernetes library version (#18797)

Rename PodLauncher to PodManager (#20576)

Removes Python 3.6 support (#20467)

Add deprecation warning for non-json-serializable params (#20174)

Rename TaskMixin to DependencyMixin (#20297)

Deprecate passing execution_date to XCom methods (#19825)

Remove get_readable_dags and get_editable_dags, and get_accessible_dags. (#19961)

Remove postgres 9.6 support (#19987)

Removed hardcoded connection types. Check if hook is instance of DbApiHook. (#19639)

add kubernetes 1.21 support (#19557)

Add FAB base class and set import_name explicitly. (#19667)

Removes unused state transitions to handle auto-changing view permissions. (#19153)

Chore: Use enum for __var and __type members (#19303)

Use fab models (#19121)

Consolidate method names between Airflow Security Manager and FAB default (#18726)

Remove distutils usages for Python 3.10 (#19064)

Removing redundant max_tis_per_query initialisation on SchedulerJob (#19020)

Remove deprecated usage of init_role() from API (#18820)

Remove duplicate code on dbapi hook (#18821)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.3.0-source.tar.gz(22.18 MB)
apache-airflow-2.3.0-source.tar.gz.asc(699 bytes)
apache-airflow-2.3.0-source.tar.gz.sha512(165 bytes)
apache-airflow-2.3.0.tar.gz(4.27 MB)
apache-airflow-2.3.0.tar.gz.asc(699 bytes)
apache-airflow-2.3.0.tar.gz.sha512(158 bytes)
apache_airflow-2.3.0-py3-none-any.whl(4.94 MB)
apache_airflow-2.3.0-py3-none-any.whl.asc(699 bytes)
apache_airflow-2.3.0-py3-none-any.whl.sha512(168 bytes)
2.2.5(Apr 4, 2022)
Bug Fixes

Check and disallow a relative path for sqlite (#22530)

Fixed dask executor and tests (#22027)

Fix broken links to celery documentation (#22364)

Fix incorrect data provided to tries & landing times charts (#21928)

Fix assignment of unassigned triggers (#21770)

Fix triggerer --capacity parameter (#21753)

Fix graph autorefresh on page load (#21736)

Fix filesystem sensor for directories (#21729)

Fix stray order_by(TaskInstance.execution_date) (#21705)

Correctly handle multiple '=' in LocalFileSystem secrets. (#21694)

Log exception in local executor (#21667)

Disable default_pool delete on web ui (#21658)

Extends typing-extensions to be installed with python 3.8+ #21566 (#21567)

Dispose unused connection pool (#21565)

Fix logging JDBC SQL error when task fails (#21540)

Filter out default configs when overrides exist. (#21539)

Fix Resources __eq__ check (#21442)

Fix max_active_runs=1 not scheduling runs when min_file_process_interval is high (#21413)

Reduce DB load incurred by Stale DAG deactivation (#21399)

Fix race condition between triggerer and scheduler (#21316)

Fix trigger dag redirect from task instance log view (#21239)

Log traceback in trigger excs (#21213)

A trigger might use a connection; make sure we mask passwords (#21207)

Update ExternalTaskSensorLink to handle templated external_dag_id (#21192)

Ensure clear_task_instances sets valid run state (#21116)

Fix: Update custom connection field processing (#20883)

Truncate stack trace to DAG user code for exceptions raised during execution (#20731)

Fix duplicate trigger creation race condition (#20699)

Fix Tasks getting stuck in scheduled state (#19747)

Fix: Do not render undefined graph edges (#19684)

Set X-Frame-Options header to DENY only if X_FRAME_ENABLED is set to true. (#19491)

Doc only changes

adding on_execute_callback to callbacks docs (#22362)

Add documentation on specifying a DB schema. (#22347)

Fix postgres part of pipeline example of tutorial (#21586)

Extend documentation for states of DAGs & tasks and update trigger rules docs (#21382)

DB upgrade is required when updating Airflow (#22061)

Remove misleading MSSQL information from the docs (#21998)

Misc

Add the new Airflow Trove Classifier to setup.cfg (#22241)

Rename to_delete to to_cancel in TriggerRunner (#20658)

Update Flask-AppBuilder to 3.4.5 (#22596)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.2.5-source.tar.gz(19.99 MB)
apache-airflow-2.2.5-source.tar.gz.asc(699 bytes)
apache-airflow-2.2.5-source.tar.gz.sha512(165 bytes)
apache-airflow-2.2.5.tar.gz(4.38 MB)
apache-airflow-2.2.5.tar.gz.asc(699 bytes)
apache-airflow-2.2.5.tar.gz.sha512(158 bytes)
apache_airflow-2.2.5-py3-none-any.whl(5.01 MB)
apache_airflow-2.2.5-py3-none-any.whl.asc(699 bytes)
apache_airflow-2.2.5-py3-none-any.whl.sha512(168 bytes)
helm-chart/1.5.0(Mar 11, 2022)
New Features

Add support for custom command and args in jobs (#20864)

Support for priorityClassName (#20794)

Add envFrom to the Flower deployment (#21401)

Add annotations to cleanup pods (#21484)

Improvements

Speedup liveness probe for scheduler and triggerer (#20833, #21108, #22041)

Update git-sync to v3.4.0 (#21309)

Remove default auth backend setting (#21640)

Bug Fixes

Fix elasticsearch URL when username/password are empty (#21222)

Mount airflow.cfg in wait-for-airflow-migrations containers (#20609)

Grant pod log reader to triggerer serviceaccount (#21111)

Doc only changes

Simplify chart docs for configuring Airflow (#21747)

Add extra information about time synchronization needed (#21685)

Fix extra containers docs (#20787)

Fix example DAGs env var in quickstart (#21990)

Misc

Use 2.2.4 as default Airflow version (#21745)

Change Redis image to bullseye (#21875)

Source code(tar.gz)
Source code(zip)
airflow-1.5.0.tgz(143.27 KB)
airflow-1.5.0.tgz.asc(870 bytes)
airflow-1.5.0.tgz.prov(5.69 KB)
airflow-1.5.0.tgz.sha512(148 bytes)
airflow-chart-1.5.0-source.tar.gz(95.42 KB)
airflow-chart-1.5.0-source.tar.gz.asc(870 bytes)
airflow-chart-1.5.0-source.tar.gz.sha512(164 bytes)
2.2.4(Feb 22, 2022)
Bug Fixes

Adding missing login provider related methods from Flask-Appbuilder (#21294)

Fix slow DAG deletion due to missing dag_id index for job table (#20282)

Add a session backend to store session data in the database (#21478)

Show task status only for running dags or only for the last finished dag (#21352)

Use compat data interval shim in log handlers (#21289)

Fix mismatch in generated run_id and logical date of DAG run (#18707)

Fix TriggerDagRunOperator extra link (#19410)

Add possibility to create user in the Remote User mode (#19963)

Avoid deadlock when rescheduling task (#21362)

Fix the incorrect scheduling time for the first run of dag (#21011)

Fix Scheduler crash when executing task instances of missing DAG (#20349)

Deferred tasks does not cancel when DAG is marked fail (#20649)

Removed duplicated dag_run join in Dag.get_task_instances() (#20591)

Avoid unintentional data loss when deleting DAGs (#20758)

Fix session usage in /rendered-k8s view (#21006)

Fix airflow dags backfill --reset-dagruns errors when run twice (#21062)

Do not set TaskInstance.max_tries in refresh_from_task (#21018)

Don't require dag_id in body in dagrun REST API endpoint (#21024)

Add Roles from Azure OAUTH Response in internal Security Manager (#20707)

Allow Viewing DagRuns and TIs if a user has DAG "read" perms (#20663)

Fix running airflow dags test <dag_id> <execution_dt> results in error when run twice (#21031)

Switch to non-vendored latest connexion library (#20910)

Bump flask-appbuilder to >=3.3.4 (#20628)

upgrade celery to 5.2.3 (#19703)

Bump croniter from <1.1 to <1.2 (#20489)

Lift off upper bound for MarkupSafe (#20113)

Avoid calling DAG.following_schedule() for TaskInstance.get_template_context() (#20486)

Fix(standalone): Remove hardcoded Webserver port (#20429)

Remove unnecssary logging in experimental API (#20356)

Un-ignore DeprecationWarning (#20322)

Deepcopying Kubernetes Secrets attributes causing issues (#20318)

Fix(dag-dependencies): fix arrow styling (#20303)

Adds retry on taskinstance retrieval lock (#20030)

Correctly send timing metrics when using dogstatsd (fix schedule_delay metric) (#19973)

Enhance multiple_outputs inference of dict typing (#19608)

Fixing ses email backend (#18042)

Pin Markupsafe until we are able to upgrade Flask/Jinja (#21664)

Doc only changes

Added explaining concept of logical date in DAG run docs (#21433)

Add note about Variable precedence with env vars (#21568)

Update error docs to include before_send option (#21275)

Augment xcom docs (#20755)

Add documentation and release policy on "latest" constraints (#21093)

Add a link to the DAG model in the Python API reference (#21060)

Added an enum param example (#20841)

Compare taskgroup and subdag (#20700)

Add note about reserved params keyword (#20640)

Improve documentation on Params (#20567)

Fix typo in MySQL Database creation code (Set up DB docs) (#20102)

Add requirements.txt description (#20048)

Clean up default_args usage in docs (#19803)

Add docker-compose explanation to conn localhost (#19076)

Update CSV ingest code for tutorial (#18960)

Adds Pendulum 1.x -> 2.x upgrade documentation (#18955)

Updating explicit arg example in TaskFlow API tutorial doc (#18907)

Adds back documentation about context usage in Python/@task (#18868)

Clean up dynamic start_date values from docs (#19607)

Docs for multiple pool slots (#20257)

Update upgrading.rst with detailed code example of how to resolve post-upgrade warning (#19993)

Misc

Deprecate some functions in the experimental API (#19931)

Deprecate smart sensors (#20151)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.2.4-source.tar.gz(19.95 MB)
apache-airflow-2.2.4-source.tar.gz.asc(870 bytes)
apache-airflow-2.2.4-source.tar.gz.sha512(165 bytes)
apache-airflow-2.2.4.tar.gz(4.37 MB)
apache-airflow-2.2.4.tar.gz.asc(870 bytes)
apache-airflow-2.2.4.tar.gz.sha512(158 bytes)
apache_airflow-2.2.4-py3-none-any.whl(5.01 MB)
apache_airflow-2.2.4-py3-none-any.whl.asc(870 bytes)
apache_airflow-2.2.4-py3-none-any.whl.sha512(168 bytes)
helm-chart/1.4.0(Jan 10, 2022)
New Features

Add type to extra secrets param (#20599)

Support elasticsearch connection scheme (#20564)

Allows to disable built-in secret variables individually (#18974)

Add support for securityContext (#18249)

Add extra containers, volumes and volume mounts for jobs (#18808)

Allow ingress multiple hostnames w/diff secrets (#18542)

PgBouncer extra volumes, volume mounts, and sslmode (#19749)

Allow specifying kerberos keytab (#19054)

Allow disabling the Helm hooks (#18776, #20018)

Add migration-wait-timeout (#20069)

Improvements

Increase default livenessProbe timeout (#20698)

Strict schema for k8s objects for values.yaml (#19181)

Remove unnecessary pod_template_file defaults (#19690)

Use built-in check-migrations command for Airflow>=2 (#19676)

Bug Fixes

Fix precedence of affinity, nodeSelector, and tolerations (#20641)

Fix chart elasticsearch default port 80 to 9200. (#20616)

Fix network policy issue for webserver and flowerui (#20199)

Use local definitions for k8s schema validation (#20544)

Add custom labels for ingresses/PVCs (#20535)

Fix extra secrets/configmaps labels (#20464)

Fix flower restarts on update (#20316)

Properly quote namespace names (#20266)

Doc only changes

Add helm dependency update step to chart INSTALL (#20702)

Reword section covering the envvar secrets (#20566)

Add "Customizing Workers" page (#20331)

Include Datadog example in production guide (#17996)

Update production Helm guide database section to use k8s secret (#19892)

Fix multiNamespaceMode docs to also cover KPO (#19879)

Clarify Helm behaviour when it comes to loading default connections (#19708)

Misc

Use 2.2.3 as default Airflow version (#20450)

Add ArtifactHUB annotations for docs and screenshots (#20558)

Add kubernetes 1.21 support (#19557)

Source code(tar.gz)
Source code(zip)
airflow-1.4.0.tgz(143.03 KB)
airflow-1.4.0.tgz.asc(870 bytes)
airflow-1.4.0.tgz.prov(8.37 KB)
airflow-1.4.0.tgz.sha512(148 bytes)
airflow-chart-1.4.0-source.tar.gz(94.91 KB)
airflow-chart-1.4.0-source.tar.gz.asc(870 bytes)
airflow-chart-1.4.0-source.tar.gz.sha512(164 bytes)
2.2.3(Dec 21, 2021)
Bug Fixes

Lazy Jinja2 context (#20217)

Exclude snowflake-sqlalchemy v1.2.5 (#20245)

Move away from legacy importlib.resources API (#19091)

Move setgid as the first command executed in forked task runner (#20040)

Fix race condition when starting DagProcessorAgent (#19935)

Limit httpx to <0.20.0 (#20218)

Log provider import errors as debug warnings (#20172)

Bump minimum required alembic version (#20153)

Fix log link in gantt view (#20121)

fixing #19028 by moving chown to use sudo (#20114)

Lift off upper bound for MarkupSafe (#20113)

Fix infinite recursion on redact log (#20039)

Fix db downgrades (#19994)

Context class handles deprecation (#19886)

Fix possible reference to undeclared variable (#19933)

Validate DagRun state is valid on assignment (#19898)

Workaround occasional deadlocks with MSSQL (#19856)

Enable task run setting to be able reinitialise (#19845)

Fix log endpoint for same task (#19672)

Cast macro datetime string inputs explicitly (#19592)

Do not crash with stacktrace when task instance is missing (#19478)

Fix log timezone in task log view (#19342) (#19401)

Fix: Add taskgroup tooltip to graph view (#19083)

Rename execution date in forms and tables (#19063)

Simplify "invalid TI state" message (#19029)

Handle case of nonexistent file when preparing file path queue (#18998)

Do not create dagruns for DAGs with import errors (#19367)

Fix field relabeling when switching between conn types (#19411)

KubernetesExecutor should default to template image if used (#19484)

Fix task instance api cannot list task instances with None state (#19487)

Fix IntegrityError in DagFileProcessor.manage_slas (#19553)

Declare data interval fields as serializable (#19616)

Relax timetable class validation (#19878)

Fix labels used to find queued KubernetesExecutor pods (#19904)

Fix moved data migration check for MySQL when replcation is used (#19999)

Doc only changes

Warn without tracebacks when example_dags are missing deps (#20295)

Deferrable operators doc clarification (#20150)

Ensure the example DAGs are all working (#19355)

Updating core example DAGs to use TaskFlow API where applicable (#18562)

Add xcom clearing behaviour on task retries (#19968)

Add a short chapter focusing on adapting secret format for connections (#19859)

Add information about supported OS-es for Apache Airflow (#19855)

Update docs to reflect that changes to the base_log_folder require updating other configs (#19793)

Disclaimer in KubernetesExecutor pod template docs (#19686)

Add upgrade note on execution_date -> run_id (#19593)

Expanding .output operator property information in TaskFlow tutorial doc (#19214)

Add example SLA DAG (#19563)

Add a proper example to patch DAG (#19465)

Add DAG file processing description to Scheduler Concepts (#18954)

Updating explicit arg example in TaskFlow API tutorial doc (#18907)

Adds back documentation about context usage in Python/@task (#18868)

Add release date for when an endpoint/field is added in the REST API (#19203)

Better pod_template_file examples (#19691)

Add decription on how you can customize image entrypoint (#18915)

Dags-in-image pod template example should not have dag mounts (#19337)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.2.3-source.tar.gz(19.96 MB)
apache-airflow-2.2.3-source.tar.gz.asc(854 bytes)
apache-airflow-2.2.3-source.tar.gz.sha512(165 bytes)
apache-airflow-2.2.3.tar.gz(4.41 MB)
apache-airflow-2.2.3.tar.gz.asc(854 bytes)
apache-airflow-2.2.3.tar.gz.sha512(158 bytes)
apache_airflow-2.2.3-py3-none-any.whl(5.06 MB)
apache_airflow-2.2.3-py3-none-any.whl.asc(854 bytes)
apache_airflow-2.2.3-py3-none-any.whl.sha512(168 bytes)
helm-chart/1.3.0(Nov 9, 2021)
New Features

Chart: Add resources for cleanup and createuser jobs (#19263)

Chart: Add labels to jobs created by cleanup pods (#19225)

Add migration job resources (#19175)

Allow custom pod annotations to all components (#18481)

Chart: Make PgBouncer cmd/args configurable (#18910)

Chart: Use python 3.7 by default; support disabling triggerer (#18920)

Improvements

Chart: Increase default liveness probe timeout (#19003)

Chart: Mount DAGs in triggerer (#18753)

Bug Fixes

Allow Airflow UI to create worker pod via Clear > Run (#18272)

Allow airflow standard images to run in openshift utilising the official helm chart #18136 (#18147)

Doc only changes

Chart: Fix extraEnvFrom examples (#19144)

Chart docs: Update webserver secret key reference configuration (#18595)

Fix helm chart links in source install guide (#18588)

Misc

Chart: Update default Airflow version to 2.2.1 (#19326)

Modernize dockerfiles builds (#19327)

Chart: Use strict k8s schemas for template validation (#19379)

Source code(tar.gz)
Source code(zip)
airflow-1.3.0.tgz(108.09 KB)
airflow-1.3.0.tgz.asc(516 bytes)
airflow-1.3.0.tgz.prov(1.22 KB)
airflow-1.3.0.tgz.sha512(148 bytes)
airflow-chart-1.3.0-source.tar.gz(60.46 KB)
airflow-chart-1.3.0-source.tar.gz.asc(516 bytes)
airflow-chart-1.3.0-source.tar.gz.sha512(164 bytes)
2.2.2(Nov 15, 2021)
Bug Fixes

Fix bug when checking for existence of a Variable (#19395)

Fix Serialization whenrelativedelta is passed as schedule_interval (#19418)

Fix moving of dangling TaskInstance rows for SQL Server (#19425)

Fix task instance modal in gantt view (#19258)

Fix serialization of Params with set data type (#19267)

Check if job object is None before calling .is_alive() (#19380)

Task should fail immediately when pod is unprocessable (#19359)

Fix downgrade for a DB Migration (#19390)

Only mark SchedulerJobs as failed, not any jobs (#19375)

Fix message on "Mark as" confirmation page (#19363)

Bugfix: Check next run exists before reading data interval (#19307)

Fix MySQL db migration with default encoding/collation (#19268)

Fix hidden tooltip position (#19261)

sqlite_default Connection has been hard-coded to /tmp, use gettempdir instead (#19255)

Fix Toggle Wrap on DAG code page (#19211)

Clarify "dag not found" error message in CLI (#19338)

Add Note to SLA regarding schedule_interval (#19173)

Use execution_date to check for existing DagRun for TriggerDagRunOperator (#18968)

Add explicit session parameter in PoolSlotsAvailableDep (#18875)

FAB still requires WTForms<3.0 (#19466)

Fix missing dagruns when catchup=True (#19528)

Doc only changes

Add missing parameter documentation for "timetable" (#19282)

Improve Kubernetes Executor docs (#19339)

Update image tag used in docker docs

Source code(tar.gz)
Source code(zip)
apache-airflow-2.2.2-source.tar.gz(19.78 MB)
apache-airflow-2.2.2-source.tar.gz.asc(516 bytes)
apache-airflow-2.2.2-source.tar.gz.sha512(165 bytes)
apache-airflow-2.2.2.tar.gz(4.43 MB)
apache-airflow-2.2.2.tar.gz.asc(516 bytes)
apache-airflow-2.2.2.tar.gz.sha512(158 bytes)
apache_airflow-2.2.2-py3-none-any.whl(5.06 MB)
apache_airflow-2.2.2-py3-none-any.whl.asc(516 bytes)
apache_airflow-2.2.2-py3-none-any.whl.sha512(168 bytes)
2.2.1(Oct 30, 2021)
Bug Fixes

Fix Unexpected commit error in schedulerjob (#19213)

Add DagRun.logical_date as a property (#19198)

Clear ti.next_method and ti.next_kwargs on task finish (#19183)

Faster PostgreSQL db migration to Airflow 2.2 (#19166)

Remove incorrect type comment in Swagger2Specification._set_defaults classmethod (#19065)

Add TriggererJob to jobs check command (#19179, #19185)

Hide tooltip when next run is None (#19112)

Crerate TI context with data interval compat layer (#19148)

Fix queued dag runs changes catchup=False behaviour (#19130, #19145)

add detailed information to logging when a dag or a task finishes. (#19097)

Warn about unsupported Python 3.10 (#19060)

Fix catchup by limiting queued dagrun creation using max_active_runs (#18897)

Prevent scheduler crash when serialized dag is missing (#19113)

Don't install SQLAlchemy/Pendulum adapters for other DBs (#18745)

Workaround libstdcpp TLS error (#19010)

Change ds, ts, etc. back to use logical date (#19088)

Ensure task state doesn't change when marked as failed/success/skipped (#19095)

Relax packaging requirement (#19087)

Rename trigger page label to Logical Date (#19061)

Allow Param to support a default value of None (#19034)

Upgrade old DAG/task param format when deserializing from the DB (#18986)

Don't bake ENV and _cmd into tmp config for non-sudo (#18772)

CLI: Fail backfill command before loading DAGs if missing args (#18994)

BugFix: Null execution date on insert to task_fail violating NOT NULL (#18979)

Try to move "dangling" rows in upgradedb (#18953)

Row lock TI query in SchedulerJob._process_executor_events (#18975)

Sentry before send fallback (#18980)

Fix XCom.delete error in Airflow 2.2.0 (#18956)

Check python version before starting triggerer (#18926)

Doc only changes

Update access control documentation for TaskInstances and DagRuns (#18644)

Add information about keepalives for managed Postgres (#18850)

Doc: Add Callbacks Section to Logging & Monitoring (#18842)

Group PATCH DAGrun together with other DAGRun endpoints (#18885)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.2.1-source.tar.gz(19.78 MB)
apache-airflow-2.2.1-source.tar.gz.asc(516 bytes)
apache-airflow-2.2.1-source.tar.gz.sha512(165 bytes)
apache-airflow-2.2.1.tar.gz(4.40 MB)
apache-airflow-2.2.1.tar.gz.asc(516 bytes)
apache-airflow-2.2.1.tar.gz.sha512(158 bytes)
apache_airflow-2.2.1-py3-none-any.whl(5.06 MB)
apache_airflow-2.2.1-py3-none-any.whl.asc(516 bytes)
apache_airflow-2.2.1-py3-none-any.whl.sha512(168 bytes)
2.2.0(Oct 11, 2021)
New Features

AIP-39: Add (customizable) Timetable class to Airflow for richer scheduling behaviour (#15397, #16030, #16352, #17030, #17122, #17414, #17552, #17755, #17989, #18084, #18088, #18244, #18266, #18420, #18434, #18421, #18475, #18499, #18573, #18522, #18729, #18706, #18742, #18786, #18804)

AIP-40: Add Deferrable "Async" Tasks (#15389, #17564, #17565, #17601, #17745, #17747, #17748, #17875, #17876, #18129, #18210, #18214, #18552, #18728, #18414)

Add a Docker Taskflow decorator (#15330, #18739)

Add Airflow Standalone command (#15826)

Display alert messages on dashboard from local settings (#18284)

Advanced Params using json-schema (#17100)

Ability to test connections from UI or API (#15795, #18750)

Add Next Run to UI (#17732)

Add default weight rule configuration option (#18627)

Add a calendar field to choose the execution date of the DAG when triggering it (#16141)

Allow setting specific cwd for BashOperator (#17751)

Show import errors in DAG views (#17818)

Add pre/post execution hooks [Experimental] (#17576)

Added table to view providers in Airflow ui under admin tab (#15385)

Adds secrets backend/logging/auth information to provider yaml (#17625)

Add date format filters to Jinja environment (#17451)

Introduce RESTARTING state (#16681)

Webserver: Unpause DAG on manual trigger (#16569)

API endpoint to create new user (#16609)

Add insert_args for support transfer replace (#15825)

Add recursive flag to glob in filesystem sensor (#16894)

Add conn to jinja template context (#16686)

Add default_args for TaskGroup (#16557)

Allow adding duplicate connections from UI (#15574)

Allow specifying multiple URLs via the CORS config option (#17941)

Implement API endpoint for DAG deletion (#17980)

Add DAG run endpoint for marking a dagrun success or failed(#17839)

Add support for kinit options [-f|-F] and [-a|-A] (#17816)

Queue support for DaskExecutor using Dask Worker Resources (#16829, #18720)

Make auto refresh interval configurable (#18107)

Improvements

Small improvements for Airflow UI (#18715, #18795)

Rename processor_poll_interval to scheduler_idle_sleep_time (#18704)

Check the allowed values for the logging level (#18651)

Fix error on triggering a dag that doesn't exist using dagrun_conf (#18655)

Add muldelete action to TaskInstanceModelView (#18438)

Avoid importing DAGs during clean DB installation (#18450)

Require can_edit on DAG privileges to modify TaskInstances and DagRuns (#16634)

Make Kubernetes job description fit on one log line (#18377)

Always draw borders if task instance state is null or undefined (#18033)

Inclusive Language (#18349)

Improved log handling for zombie tasks (#18277)

Adding Variable.update method and improving detection of variable key collisions (#18159)

Add note about params on trigger DAG page (#18166)

Change TaskInstance and TaskReschedule PK from execution_date to run_id (#17719)

Adding TaskGroup support in BaseOperator.chain() (#17456)

Advises the kernel to not cache log files generated by Airflow (#18054)

Allow filtering DAGS by tags in the REST API (#18090)

Optimize imports of Providers Manager (#18052)

Adds capability of Warnings for incompatible community providers (#18020)

Serialize the template_ext attribute to show it in UI (#17985)

Add robots.txt and X-Robots-Tag header (#17946)

Refactor BranchDayOfWeekOperator, DayOfWeekSensor (#17940)

Update error message to guide the user into self-help mostly (#17929)

Update to Celery 5 (#17397)

Avoid redirect loop for users with no permissions (#17838)

Improve graph view load time for dags with open groups (#17821)

Add links to provider's documentation (#17736)

Remove Marshmallow schema warnings (#17753)

Rename none_failed_or_skipped by none_failed_min_one_success trigger rule (#17683)

Remove [core] store_dag_code & use DB to get Dag Code (#16342)

Rename task_concurrency to max_active_tis_per_dag (#17708)

Import Hooks lazily individually in providers manager (#17682)

Adding support for multiple task-ids in the external task sensor (#17339)

Replace execution_date with run_id in airflow tasks run command (#16666)

Avoid endless redirect loop when user has no roles (#17613)

Make output from users cli command more consistent (#17642)

Open relative extra links in place (#17477)

Move worker_log_server_port option to the logging section (#17621)

Use gunicorn to serve logs generated by worker (#17591)

Improve validation of Group id (#17578)

Simplify 404 page (#17501)

Add XCom.clear so it's hookable in custom XCom backend (#17405)

Add deprecation notice for SubDagOperator (#17488)

Support DAGS folder being in different location on scheduler and runners (#16860)

Remove /dagrun/create and disable edit form generated by F.A.B (#17376)

Enable specifying dictionary paths in template_fields_renderers (#17321)

error early if virtualenv is missing (#15788)

Handle connection parameters added to Extra and custom fields (#17269)

Fix airflow celery stop to accept the pid file. (#17278)

Remove DAG refresh buttons (#17263)

Deprecate dummy trigger rule in favor of always (#17144)

Be verbose about failure to import airflow_local_settings (#17195)

Include exit code in AirflowException str when BashOperator fails. (#17151)

Adding EdgeModifier support for chain() (#17099)

Only allows supported field types to be used in custom connections (#17194)

Secrets backend failover (#16404)

Warn on Webserver when using SQLite or SequentialExecutor (#17133)

Extend init_containers defined in pod_override (#17537)

Client-side filter dag dependencies (#16253)

Improve executor validation in CLI (#17071)

Prevent running airflow db init/upgrade migrations and setup in parallel. (#17078)

Update chain() and cross_downstream() to support XComArgs (#16732)

Improve graph view refresh (#16696)

When a task instance fails with exception, log it (#16805)

Set process title for serve-logs and LocalExecutor (#16644)

Rename test_cycle to check_cycle (#16617)

Set process title for Worker when using LocalExecutor (#16623)

Add schema as DbApiHook instance attribute (#16521, #17423)

Improve compatibility with MSSQL (#9973)

Add transparency for unsupported connection type (#16220)

Call resource based fab methods (#16190)

Format more dates with timezone (#16129)

Replace deprecated dag.sub_dag with dag.partial_subset (#16179)

Treat AirflowSensorTimeout as immediate failure without retrying (#12058)

Marking success/failed automatically clears failed downstream tasks (#13037)

Add close/open indicator for import dag errors (#16073)

Add collapsible import errors (#16072)

Always return a response in TI's action_clear view (#15980)

Add cli command to delete user by email (#15873)

Use resource and action names for FAB permissions (#16410)

Rename DAG concurrency ([core] dag_concurrency) settings for easier understanding (#16267, #18730)

Calendar UI improvements (#16226)

Refactor: SKIPPED should not be logged again as SUCCESS (#14822)

Remove version limits for dnspython (#18046, #18162)

Accept custom run ID in TriggerDagRunOperator (#18788)

Bug Fixes

Make REST API patch user endpoint work the same way as the UI (#18757)

Properly set start_date for cleared tasks (#18708)

Ensure task_instance exists before running update on its state(REST API) (#18642)

Make AirflowDateTimePickerWidget a required field (#18602)

Retry deadlocked transactions on deleting old rendered task fields (#18616)

Fix retry_exponential_backoff divide by zero error when retry delay is zero (#17003)

Improve how UI handles datetimes (#18611, #18700)

Bugfix: dag_bag.get_dag should return None, not raise exception (#18554)

Only show the task modal if it is a valid instance (#18570)

Fix accessing rendered {{ task.x }} attributes from within templates (#18516)

Add missing email type of connection (#18502)

Don't use flash for "same-page" UI messages. (#18462)

Fix task group tooltip (#18406)

Properly fix dagrun update state endpoint (#18370)

Properly handle ti state difference between executor and scheduler (#17819)

Fix stuck "queued" tasks in KubernetesExecutor (#18152)

Don't permanently add zip DAGs to sys.path (#18384)

Fix random deadlocks in MSSQL database (#18362)

Deactivating DAGs which have been removed from files (#17121)

When syncing dags to db remove dag_tag rows that are now unused (#8231)

Graceful scheduler shutdown on error (#18092)

Fix mini scheduler not respecting wait_for_downstream dep (#18338)

Pass exception to run_finished_callback for Debug Executor (#17983)

Make XCom.get_one return full, not abbreviated values (#18274)

Use try/except when closing temporary file in task_runner (#18269)

show next run if not none (#18273)

Fix DB session handling in XCom.set (#18240)

Sort adopted tasks in _check_for_stalled_adopted_tasks method (#18208)

Fix external_executor_id not being set for manually run jobs (#17207)

Fix deleting of zipped Dags in Serialized Dag Table (#18243)

Return explicit error on user-add for duplicated email (#18224)

Remove loading dots even when last run data is empty (#18230)

Swap dag import error dropdown icons (#18207)

Automatically create section when migrating config (#16814)

Set encoding to utf-8 by default while reading task logs (#17965)

Set task state to failed when pod is DELETED while running (#18095)

Apply parent dag permissions to subdags (#18160)

Change id collation for MySQL to case-sensitive (#18072)

Logs task launch exception in StandardTaskRunner (#17967)

Applied permissions to self._error_file (#15947)

Hide variable import form if user lacks permission (#18000)

Fix blank dag dependencies view (#17990)

Add missing menu access for dag dependencies and configurations pages (#17450)

Fix passing Jinja templates in DateTimeSensor (#17959)

Fixing bug which restricted the visibility of ImportErrors (#17924)

Fix grammar in traceback.html (#17942)

Fix DagRunState enum query for MySQLdb driver (#17886)

Fixed button size in "Actions" group. (#17902)

Fix log links on graph TI modal (#17862)

Only show import errors for DAGs a user can access (#17835)

Show all import_errors from zip files (#17759)

fix EXTRA_LOGGER_NAMES param and related docs (#17808)

Use one interpreter for Airflow and gunicorn (#17805)

Fix: Mysql 5.7 id utf8mb3 (#14535)

Fix dag_processing.last_duration metric random holes (#17769)

Automatically use utf8mb3_general_ci collation for MySQL (#17729)

Fix wrong query on running tis (#17631)

Add root to tree refresh url (#17633)

Do not delete running DAG from the UI (#17630)

Rescue if a DagRun's DAG was removed from db (#17544)

fix: filter condition of TaskInstance does not work #17535 (#17548)

Regression on pid reset to allow task start after heartbeat (#17333)

Dont use TaskInstance in CeleryExecutor.trigger_tasks (#16248)

Remove locks for upgrades in MSSQL (#17213)

Create virtualenv via python call (#17156)

Ensure a DAG is acyclic when running DAG.cli() (#17105)

Translate non-ascii characters (#17057)

Change the logic of None comparison in model_list template (#16893)

Have UI and POST /task_instances_state API endpoint have same behaviour (#16539)

ensure task is skipped if missing sla (#16719)

Fix direct use of cached_property module (#16710)

Fix TI success confirm page (#16650)

Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173)

Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989)

Modify return value check in python virtualenv jinja template (#16049)

Fix dag dependency search (#15924)

Make custom JSON encoder support Decimal (#16383)

Bugfix: Allow clearing tasks with just dag_id and empty subdir (#16513)

Convert port value to a number before calling test connection (#16497)

Handle missing/null serialized DAG dependencies (#16393)

Correctly set dag.fileloc when using the @dag decorator (#16384)

Fix TI success/failure links (#16233)

Correctly implement autocomplete early return in airflow/www/views.py (#15940)

Backport fix to allow pickling of Loggers to Python 3.6 (#18798)

Fix bug that Backfill job fail to run when there are tasks run into reschedule state (#17305, #18806)

Doc only changes

Update dagbag_size documentation (#18824)

Update documentation about bundle extras (#18828)

Fix wrong Postgres search_path set up instructions (#17600)

Remove AIRFLOW_GID from Docker images (#18747)

Improve error message for BranchPythonOperator when no task_id to follow (#18471)

Improve guidance to users telling them what to do on import timeout (#18478)

Explain scheduler fine-tuning better (#18356)

Added example JSON for airflow pools import (#18376)

Add sla_miss_callback section to the documentation (#18305)

Explain sentry default environment variable for subprocess hook (#18346)

Refactor installation pages (#18282)

Improves installing from sources pages for all components (#18251)

Doc: Use closer.lua script for downloading sources (#18179)

Improves quick-start docker-compose warnings and documentation (#18164)

Production-level support for MSSQL (#18382)

Update non-working example in documentation (#18067)

Improve the description of how to handle dynamic task generation (#17963)

Remove default_args pattern + added get_current_context() use for Core Airflow example DAGs (#16866)

Update max_tis_per_query to better render on the webpage (#17971)

Adds Github Oauth example with team based authorization (#17896)

Update docker.rst (#17882)

Example xcom update (#17749)

Add doc warning about connections added via envvars (#17915)

fix wrong documents around upgrade-check.rst (#17903)

Add Brent to Committers list (#17873)

Improves documentation about modules management (#17757)

Remove deprecated metrics from metrics.rst (#17772)

Make sure "production-readiness" of docker-compose is well explained (#17731)

Doc: Update Upgrade to v2 docs with Airflow 1.10.x EOL dates (#17710)

Doc: Replace deprecated param from docstrings (#17709)

docs(dagowner): describe dag owner more carefully (#17699)

docs(impersonation): update note so avoid misintrepretation (#17701)

Docs: Make DAG.is_active read-only in API (#17667)

Improve cross-links to operators and hooks references (#17622)

Update documentation regarding Python 3.9 support (#17611)

Fix MySQL database character set instruction (#17603)

Document overriding XCom.clear for data lifecycle management (#17589)

Path correction in docs for airflow core (#17567)

docs(celery): reworded, add actual multiple queues example (#17541)

Doc: Add FAQ to speed up parsing with tons of dag files (#17519)

Improve image building documentation for new users (#17409)

Doc: Strip unnecessary arguments from MariaDB JIRA URL (#17296)

Update warning about MariaDB and multiple schedulers (#17287)

Doc: Recommend using same configs on all Airflow components (#17146)

Move docs about masking to a new page (#17007)

Docs: Suggest use of Env vars instead of Airflow Vars in best practises doc (#16926)

Docs: Better description for pod_template_file (#16861)

Add Aneesh Joseph as Airflow Committer (#16835)

Docs: Added new pipeline example for the tutorial docs (#16548)

Remove upstart from docs (#16672)

Add new committers: Jed and TP (#16671)

Docs: Fix flask-ouathlib to flask-oauthlib in Upgrading docs (#16320)

Docs: Fix creating a connection docs (#16312)

Docs: Fix url for Elasticsearch (#16275)

Small improvements for README.md files (#16244)

Fix docs for dag_concurrency (#16177)

Check syntatic correctness for code-snippets (#16005)

Improvements for Docker Image docs (#14843)

Add proper link for wheel packages in docs. (#15999)

Add Docs for default_pool slots (#15997)

Add memory usage warning in quick-start documentation (#15967)

Update example KubernetesExecutor git-sync pod template file (#15904)

Docs: Fix Taskflow API docs (#16574)

Added new pipeline example for the tutorial docs (#16084)

Updating the DAG docstring to include render_template_as_native_obj (#16534)

Update docs on setting up SMTP (#16523)

Docs: Fix API verb from POST to PATCH (#16511)

Misc/Internal

Renaming variables to be consistent with code logic (#18685)

Simplify strings previously split across lines (#18679)

fix exception string of BranchPythonOperator (#18623)

Add multiple roles when creating users (#18617)

Move FABs base Security Manager into Airflow. (#16647)

Remove unnecessary css state colors (#18461)

Update boto3 to <1.19 (#18389)

Improve coverage for airflow.security.kerberos module (#18258)

Fix kinesis test (#18337)

Fix provider test acessing importlib-resources (#18228)

Silence warnings in tests from using SubDagOperator (#18275)

Fix usage of range(len()) to enumerate (#18174)

Test coverage on the autocomplete view (#15943)

Update version added fields in airflow/config_templates/config.yml (#18128)

Add "packaging" to core requirements (#18122)

Adds LoggingMixins to BaseTrigger (#18106)

Limit colorlog version (6.x is incompatible) (#18099)

Fix building docs in main builds (#18035)

Remove upper-limit on tenacity (#17593)

Remove redundant numpy dependency (#17594)

Bump mysql-connector-python to latest version (#17596)

Make pandas an optional core dependency (#17575)

Add more typing to airflow.utils.helpers (#15582)

Chore: Some code cleanup in airflow/utils/db.py (#17090)

Refactor: Remove processor_factory from DAG processing (#16659)

Remove AbstractDagFileProcessorProcess from dag processing (#16816)

Update TaskGroup typing (#16811)

Update click to 8.x (#16779)

Remove remaining Pylint disables (#16760)

Remove duplicated try, there is already a try in create_session (#16701)

Removes pylint from our toolchain (#16682)

Refactor usage of unneeded function call (#16653)

Add type annotations to setup.py (#16658)

Remove SQLAlchemy <1.4 constraint (#16630) (Note: our dependencies still have a requirement on <1.4)

Refactor dag.clear method (#16086)

Use DAG_ACTIONS constant (#16232)

Use updated _get_all_non_dag_permissions method (#16317)

Add updated-name wrappers for built-in FAB methods (#16077)

Pins docutils to <0.17 until breaking behaviour is fixed (#16133)

Remove TaskInstance.log_filepath attribute (#15217)

Removes unnecessary function call in airflow/www/app.py (#15956)

Move plyvel to google provider extra (#15812)

Update permission migrations to use new naming scheme (#16400)

Use resource and action names for FAB (#16380)

Swap out calls to find_permission_view_menu for get_permission wrapper (#16377)

Fix deprecated default for fab_logging_level to WARNING (#18783)

Allow running tasks from UI when using CeleryKubernetesExecutor (#18441)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.2.0-source.tar.gz(19.77 MB)
apache-airflow-2.2.0-source.tar.gz.asc(516 bytes)
apache-airflow-2.2.0-source.tar.gz.sha512(165 bytes)
apache-airflow-2.2.0.tar.gz(4.40 MB)
apache-airflow-2.2.0.tar.gz.asc(516 bytes)
apache-airflow-2.2.0.tar.gz.sha512(158 bytes)
apache_airflow-2.2.0-py3-none-any.whl(5.05 MB)
apache_airflow-2.2.0-py3-none-any.whl.asc(516 bytes)
apache_airflow-2.2.0-py3-none-any.whl.sha512(168 bytes)
helm-chart/1.2.0(Sep 28, 2021)
New Features

Add Triggerer to Helm Chart (#17743)

Chart: warn when webserver secret key isn't set (#18306)

add extraContainers for migrateDatabaseJob (#18379)

Labels on job templates (#18403)

Chart: Allow running and waiting for DB Migrations using default image (#18218)

Chart: Make cleanup cronjob cmd/args configuable (#17970)

Chart: configurable number of retention days for log groomers (#17764)

Chart: Add loadBalancerSourceRanges in webserver and flower services (#17666)

Chart: Support extraContainers in k8s workers (#17562)

Improvements

Switch to latest version of PGBouncer-Exporter (#18429)

Chart: Ability to access http k8s via multiple hostnames (#18257)

Chart: Use stable API versions where available (#17211)

Chart: Allow podTemplate to be templated (#17560)

Bug Fixes

Chart: Fix applying labels on Triggerer (#18299)

Fixes warm shutdown for celery worker. (#18068)

Chart: Fix minor Triggerer issues (#18105)

Chart: fix webserver secret key update (#18079)

Chart: fix running with uid 0 (#17688)

Chart: use serviceaccount template for log reader rolebinding (#17645)

Chart: Fix elasticsearch-secret template port default function (#17428)

KEDA task count query should ignore k8s queue (#17433)

Doc only changes

Chart Doc: Delete extra space in adding connections doc (#18424)

Improves installing from sources pages for all components (#18251)

Chart docs: Format loadBalancerSourceRanges using code-block (#17763)

Doc: Fix a broken link in an ssh-related warning message (#17294)

Chart: Add instructions to Update Helm Repo before upgrade (#17282)

Chart docs: better note for logs existing pvc permissions (#17177)

Misc

Chart: Update the default Airflow version to 2.1.4 (#18354)

Source code(tar.gz)
Source code(zip)
airflow-1.2.0.tgz(106.76 KB)
airflow-1.2.0.tgz.asc(516 bytes)
airflow-1.2.0.tgz.prov(1.22 KB)
airflow-1.2.0.tgz.sha512(148 bytes)
airflow-chart-1.2.0-source.tar.gz(59.21 KB)
airflow-chart-1.2.0-source.tar.gz.asc(516 bytes)
airflow-chart-1.2.0-source.tar.gz.sha512(164 bytes)
2.1.4(Sep 18, 2021)
Bug Fixes

Fix deprecation error message rather than silencing it (#18126)

Limit the number of queued dagruns created by the Scheduler (#18065)

Fix DagRun execution order from queued to running not being properly followed (#18061)

Fix max_active_runs not allowing moving of queued dagruns to running (#17945)

Avoid redirect loop for users with no permissions (#17838)

Avoid endless redirect loop when user has no roles (#17613)

Fix log links on graph TI modal (#17862)

Hide variable import form if user lacks permission (#18000)

Improve dag/task concurrency check (#17786)

Fix Clear task instances endpoint resets all DAG runs bug (#17961)

Fixes incorrect parameter passed to views (#18083) (#18085)

Fix Sentry handler from LocalTaskJob causing error (#18119)

Limit colorlog version (6.x is incompatible) (#18099)

Only show Pause/Unpause tooltip on hover (#17957)

Improve graph view load time for dags with open groups (#17821)

Increase width for Run column (#17817)

Fix wrong query on running tis (#17631)

Add root to tree refresh url (#17633)

Do not delete running DAG from the UI (#17630)

Improve discoverability of Provider packages' functionality

Do not let create_dagrun overwrite explicit run_id (#17728)

BugFix: Regression on pid reset to allow task start after heartbeat (#17333)

Set task state to failed when pod is DELETED while running (#18095)

Advises the kernel to not cache log files generated by Airflow (#18054)

Sort adopted tasks in _check_for_stalled_adopted_tasks method (#18208)

Fix DagRunState enum query for MySQLdb driver (#17886)

Doc only changes

Update version added fields in airflow/config_templates/config.yml (#18128)

Improve the description of how to handle dynamic task generation (#17963)

Improve cross-links to operators and hooks references (#17622)

Doc: Fix replacing Airflow version for Docker stack (#17711)

Make the providers operators/hooks reference much more usable (#17768)

Update description about the new connection-types provider meta-data

Suggest to use secrets backend for variable when it contains sensitive data (#17319)

Improves quick-start docker-compose warnings and documentation (#18164)

Separate Installing from sources section and add more details (#18171)

Doc: Use closer.lua script for downloading sources (#18179)

Doc: Improve installing from sources (#18194)

Improves installing from sources pages for all components (#18251)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.1.4-source.tar.gz(19.23 MB)
apache-airflow-2.1.4-source.tar.gz.asc(516 bytes)
apache-airflow-2.1.4-source.tar.gz.sha512(165 bytes)
apache-airflow-2.1.4.tar.gz(4.37 MB)
apache-airflow-2.1.4.tar.gz.asc(516 bytes)
apache-airflow-2.1.4.tar.gz.sha512(158 bytes)
apache_airflow-2.1.4-py3-none-any.whl(5.01 MB)
apache_airflow-2.1.4-py3-none-any.whl.asc(516 bytes)
apache_airflow-2.1.4-py3-none-any.whl.sha512(168 bytes)
2.1.3(Sep 11, 2021)
Bug Fixes

Fix task retries when they receive sigkill and have retries and properly handle sigterm (#16301)

Fix redacting secrets in context exceptions. (#17618)

Fix race condition with dagrun callbacks (#16741)

Add 'queued' to DagRunState (#16854)

Add 'queued' state to DagRun (#16401)

Fix external elasticsearch logs link (#16357)

Add proper warning message when recorded PID is different from current PID (#17411)

Fix running tasks with default_impersonation config (#17229)

Rescue if a DagRun's DAG was removed from db (#17544)

Fixed broken json_client (#17529)

Handle and log exceptions raised during task callback (#17347)

Fix CLI kubernetes cleanup-pods which fails on invalid label key (#17298)

Show serialization exceptions in DAG parsing log (#17277)

Fix: TaskInstance does not show queued_by_job_id & external_executor_id (#17179)

Adds more explanatory message when SecretsMasker is not configured (#17101)

Enable the use of __init_subclass__ in subclasses of BaseOperator (#17027)

Fix task instance retrieval in XCom view (#16923)

Validate type of priority_weight during parsing (#16765)

Correctly handle custom deps` andtask_group`` during DAG Serialization (#16734)

Fix slow (cleared) tasks being be adopted by Celery worker. (#16718)

Fix calculating duration in tree view (#16695)

Fix AttributeError: datetime.timezone object has no attribute name (#16599)

Redact conn secrets in webserver logs (#16579)

Change graph focus to top of view instead of center (#16484)

Fail tasks in scheduler when executor reports they failed (#15929)

fix(smart_sensor): Unbound variable errors (#14774)

Add back missing permissions to UserModelView controls. (#17431)

Better diagnostics and self-healing of docker-compose (#17484)

Improve diagnostics message when users have secret_key mis-configured (#17410)

Stop checking execution_date in task_instance.refresh_from_db (#16809)

Improvements

Run mini scheduler in LocalTaskJob during task exit (#16289)

Remove SQLAlchemy<1.4 constraint (#16630)

Bump Jinja2 upper-bound from 2.12.0 to 4.0.0 (#16595)

Bump dnspython (#16698)

Updates to FlaskAppBuilder 3.3.2+ (#17208)

Add State types for tasks and DAGs (#15285)

Set Process title for Worker when using LocalExecutor (#16623)

Move DagFileProcessor and DagFileProcessorProcess out of scheduler_job.py (#16581)

Doc-only changes

Fix inconsistencies in configuration docs (#17317)

Fix docs link for using SQLite as Metadata DB (#17308)

Misc

Switch back http provider after requests removes LGPL dependency (#16974)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.1.3-source.tar.gz(19.21 MB)
apache-airflow-2.1.3-source.tar.gz.asc(516 bytes)
apache-airflow-2.1.3-source.tar.gz.sha512(165 bytes)
apache-airflow-2.1.3.tar.gz(4.36 MB)
apache-airflow-2.1.3.tar.gz.asc(516 bytes)
apache-airflow-2.1.3.tar.gz.sha512(158 bytes)
apache_airflow-2.1.3-py3-none-any.whl(5.01 MB)
apache_airflow-2.1.3-py3-none-any.whl.asc(516 bytes)
apache_airflow-2.1.3-py3-none-any.whl.sha512(168 bytes)
helm-chart/1.1.0(Jul 28, 2021)
New Features

Chart: Allow using krb5.conf with CeleryExecutor (#16822)

Chart: Refactor webserver and flower networkpolicy (#16619)

Chart: Apply worker's node assigning settings to Pod Template File (#16663)

Chart: Support for overriding webserver and flower service ports (#16572)

Chart: Support extraContainers and extraVolumes in flower (#16515)

Chart: Allow configuration of pod resources in helm chart (#16425)

Chart: Support job level annotations; fix jobs scheduling config (#16331)

feat: Helm chart adding minReplicaCount to the Keda worker-kedaautoscaler.yaml (#16262)

Chart: Adds support for custom command and args (#16153)

Chart: Add extra ini config to pgbouncer (#16120)

Chart: Add extraInitContainers to scheduler/webserver/workers (#16098)

Configurable resources for git-sync sidecar (#16080)

Chart: Template airflowLocalSettings and webserver.webserverConfig (#16074)

Support strategy/updateStrategy on scheduler (#16069)

Chart: Add both airflow and extra annotations to jobs (#16058)

loadBalancerIP and annotations for both Flower and Webserver (#15972)

Improvements

Chart: Update Postgres subchart to 10.5.3 (#17041)

Chart: Update the default Airflow version to 2.1.2 (#17013)

Update default image as 2.1.1 for Helm Chart (#16785)

Chart: warn when using default logging with KubernetesExecutor (#16784)

Drop support for Helm 2 (#16575)

Chart: podAntiAffinity for scheduler, webserver, and workers (#16315)

Chart: Update the default Airflow Version to 2.1.0 (#16273)

Chart: Only mount DAGs in webserver when required (#16229)

Chart: Remove git-sync: root and dest params (#15955)

Chart: Add warning about missing knownHosts (#15950)

Bug Fixes

Chart: Create a random secret for Webserver's flask secret key (#17142)

Chart: fix labels on cleanup serviceaccount (#16722)

Chart: Fix overriding node assigning settings on Worker Deployment (#16670)

Chart: Always deploy a gitsync init container (#16339)

Chart: Fix updating from KubernetesExecutor to CeleryExecutor (#16242)

Chart: Adds labels to Kubernetes worker pods (#16203)

Chart: Allow webserver.base_url to be templated (#16126)

Chart: Fix PgBouncer exporter sidecar (#16099)

Remove dags.gitSync.excludeWebserver from chart values.schema.json (#16070)

Chart: Fix Elasticsearch secret created without Elasticsearch enabled (#16015)

Handle special characters in password sfor Helm Chart (#16004)

Fix flower serviceAccount created without flower enable (#16011)

Chart: gitsync Clean Up for KubernetesExecutor (#15925)

Mount DAGs read only when using gitsync (#15953)

Doc only changes

Chart docs: note uid write permissions for existing pvc (#17170)

Chart Docs: Add single-line description for multiNamespaceMode (#17147)

Chart: Update description for Helm chart to include 'official' (#17040)

Chart: Better comment and example for podTemplate (#16859)

Chart: Add more clear docs for setting pod_template_file.yaml (#16632)

Fix description on scheduler.livenessprobe.periodSeconds (#16486)

Chart docs: Fix extrasecrets example (#16305)

Small improvements for README.md files (#16244)

Misc

Removes pylint from our toolchain (#16682)

Update link to match what is in pre-commit (#16408)

Chart: Update the appVersion to 2.1.0 in Chart.yaml (#16337)

Rename the main branch of the Airflow repo to be main (#16149)

Update Chart version to 1.1.0-rc1 (#16124)

Source code(tar.gz)
Source code(zip)
airflow-1.1.0.tgz(102.21 KB)
airflow-1.1.0.tgz.asc(516 bytes)
airflow-1.1.0.tgz.prov(1.22 KB)
airflow-1.1.0.tgz.sha512(148 bytes)
airflow-chart-1.1.0-source.tar.gz(54.55 KB)
airflow-chart-1.1.0-source.tar.gz.asc(516 bytes)
airflow-chart-1.1.0-source.tar.gz.sha512(164 bytes)
2.1.2(Jul 14, 2021)
Bug Fixes

Only allow the webserver to request from the worker log server (#16754)

Fix "Invalid JSON configuration, must be a dict" bug (#16648)

Fix CeleryKubernetesExecutor (#16700)

Mask value if the key is token (#16474)

Fix impersonation issue with LocalTaskJob (#16852)

Misc

Add Python 3.9 support (#15515)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.1.2-source.tar.gz(19.55 MB)
apache-airflow-2.1.2-source.tar.gz.asc(516 bytes)
apache-airflow-2.1.2-source.tar.gz.sha512(165 bytes)
apache-airflow-2.1.2.tar.gz(4.36 MB)
apache-airflow-2.1.2.tar.gz.asc(516 bytes)
apache-airflow-2.1.2.tar.gz.sha512(158 bytes)
apache_airflow-2.1.2-py3-none-any.whl(5.00 MB)
apache_airflow-2.1.2-py3-none-any.whl.asc(516 bytes)
apache_airflow-2.1.2-py3-none-any.whl.sha512(168 bytes)
2.1.1(Jul 2, 2021)
Bug Fixes

Don't crash attempting to mask secrets in dict with non-string keys (#16601)

Always install sphinx_airflow_theme from PyPI (#16594)

Remove limitation for elasticsearch library (#16553)

Adding extra requirements for build and runtime of the PROD image. (#16170)

Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173)

Removes unnecessary packages from setup_requires (#16139)

Pins docutils to <0.17 until breaking behaviour is fixed (#16133)

Improvements for Docker Image docs (#14843)

Ensure that dag_run.conf is a dict (#15057)

Fix CLI connections import and migrate logic from secrets to Connection model (#15425)

Fix Dag Details start date bug (#16206)

Fix DAG run state not updated while DAG is paused (#16343)

Allow null value for operator field in task_instance schema(REST API) (#16516)

Avoid recursion going too deep when redacting logs (#16491)

Backfill: Don't create a DagRun if no tasks match task regex (#16461)

Tree View UI for larger DAGs & more consistent spacing in Tree View (#16522)

Correctly handle None returns from Query.scalar() (#16345)

Adding only_active parameter to /dags endpoint (#14306)

Don't show stale Serialized DAGs if they are deleted in DB (#16368)

Make REST API List DAGs endpoint consistent with UI/CLI behaviour (#16318)

Support remote logging in elasticsearch with filebeat 7 (#14625)

Queue tasks with higher priority and earlier execution_date first. (#15210)

Make task ID on legend have enough width and width of line chart to be 100%. (#15915)

Fix normalize-url vulnerability (#16375)

Validate retries value on init for better errors (#16415)

add num_runs query param for tree refresh (#16437)

Fix templated default/example values in config ref docs (#16442)

Add passphrase and private_key to default sensitive field names (#16392)

Fix tasks in an infinite slots pool were never scheduled (#15247)

Fix Orphaned tasks stuck in CeleryExecutor as running (#16550)

Don't fail to log if we can't redact something (#16118)

Set max tree width to 1200 pixels (#16067)

Fill the "job_id" field for airflow task run without --local/--raw for KubeExecutor (#16108)

Fixes problem where conf variable was used before initialization (#16088)

Fix apply defaults for task decorator (#16085)

Parse recently modified files even if just parsed (#16075)

Ensure that we don't try to mask empty string in logs (#16057)

Don't die when masking log.exception when there is no exception (#16047)

Restores apply_defaults import in base_sensor_operator (#16040)

Fix auto-refresh in tree view When webserver ui is not in / (#16018)

Fix dag.clear() to set multiple dags to running when necessary (#15382)

Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989)

Source code(tar.gz)
Source code(zip)
apache-airflow-2.1.1-source.tar.gz(19.57 MB)
apache-airflow-2.1.1-source.tar.gz.asc(854 bytes)
apache-airflow-2.1.1-source.tar.gz.sha512(165 bytes)
apache-airflow-2.1.1.tar.gz(4.30 MB)
apache-airflow-2.1.1.tar.gz.asc(854 bytes)
apache-airflow-2.1.1.tar.gz.sha512(158 bytes)
apache_airflow-2.1.1-py3-none-any.whl(4.92 MB)
apache_airflow-2.1.1-py3-none-any.whl.asc(854 bytes)
apache_airflow-2.1.1-py3-none-any.whl.sha512(168 bytes)
upgrade-check/1.4.0(Jun 26, 2021)
Add conf not importable from airflow rule (#14400)

Upgrade rule to suggest rename [scheduler] max_threads to [scheduler] parsing_processes (#14913)

Fix running "upgrade_check" command in a PTY. (#14977)

Skip DatabaseVersionCheckRule check if invalid version is detected (#15122)

Fix too specific parsing of False in LegacyUIDeprecated (#14967)

Fix false positives when inheriting classes that inherit DbApiHook (#16543)

Source code(tar.gz)
Source code(zip)
apache-airflow-upgrade-check-1.4.0-bin.tar.gz(39.91 KB)
apache-airflow-upgrade-check-1.4.0-bin.tar.gz.asc(516 bytes)
apache-airflow-upgrade-check-1.4.0-bin.tar.gz.sha512(176 bytes)
apache-airflow-upgrade-check-1.4.0-source.tar.gz(39.08 KB)
apache-airflow-upgrade-check-1.4.0-source.tar.gz.asc(516 bytes)
apache-airflow-upgrade-check-1.4.0-source.tar.gz.sha512(179 bytes)
apache_airflow_upgrade_check-1.4.0-py2.py3-none-any.whl(56.73 KB)
apache_airflow_upgrade_check-1.4.0-py2.py3-none-any.whl.asc(516 bytes)
apache_airflow_upgrade_check-1.4.0-py2.py3-none-any.whl.sha512(186 bytes)
helm-chart/1.0.0(May 20, 2021)

This is the first release of the Official Helm Chart.

Docs: https://airflow.apache.org/docs/helm-chart/1.0.0/
Source code(tar.gz)
Source code(zip)
airflow-1.0.0.tgz(67.55 KB)
airflow-1.0.0.tgz.asc(516 bytes)
airflow-1.0.0.tgz.prov(1.19 KB)
airflow-1.0.0.tgz.sha512(148 bytes)
airflow-chart-1.0.0-source.tar.gz(47.01 KB)
airflow-chart-1.0.0-source.tar.gz.asc(516 bytes)
airflow-chart-1.0.0-source.tar.gz.sha512(164 bytes)
2.1.0rc2(May 18, 2021)

Source code(tar.gz)
Source code(zip)

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Related tags

Overview

Apache Airflow

Project Focus

Principles

Requirements

Getting started

Installing from PyPI

Official source code

Convenience packages

User Interface

Semantic versioning

Version Life Cycle

Support for Python and Kubernetes versions

Additional notes on Python version requirements

Contributing

Who uses Apache Airflow?

Who Maintains Apache Airflow?

Can I use the Apache Airflow logo in my presentation?

Airflow merchandise

Links

Sponsors

Comments

Why Mypy re-enable

How does it work

Can I help with the effort, you ask?

How can I help?

How can I help more ?

Current list of mypy PRs:

Remaining packages

Committer

JIRA

Description

Tests

Commits

JIRA

Description

Tests

Commits

Jira

Description

Tests

Commits

Documentation

Code Quality

Apache Airflow version

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

What happened

What you expected to happen

How to reproduce

Anything else

Are you willing to submit PR?

Code of Conduct

Apache Airflow version

What happened

What you think should happen instead

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else

Are you willing to submit PR?

Code of Conduct

Apache Airflow version

What happened

What you think should happen instead

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else

Are you willing to submit PR?

Code of Conduct

Apache Airflow version