Software:Apache Airflow

From HandWiki
Apache Airflow
Apache Airflow logo
Original author(s)Maxime Beauchemin / Airbnb
Developer(s)Apache Software Foundation
Initial releaseJune 3, 2015; 10 years ago (2015-06-03)
Written inPython
Operating systemLinux, macOS
TypeWorkflow management platform
LicenseApache License 2.0
Websiteairflow.apache.org

Apache Airflow is an open-source workflow management platform for data engineering pipelines. It started at Airbnb in October 2014[1] as a solution to manage the company's increasingly complex workflows. Creating Airflow allowed Airbnb to programmatically author and schedule their workflows and monitor them via the built-in Airflow user interface.[2][3] From the beginning, the project was made open source, becoming an Apache Incubator project in March 2016 and a top-level Apache Software Foundation project in January 2019.

Airflow is written in Python, and workflows are created via Python scripts. Airflow is designed under the principle of "configuration as code". While other "configuration as code" workflow platforms exist using markup languages like XML, using Python allows developers to import libraries and classes to help them create their workflows.

According to VentureBeat in 2025, Airflow is the de facto tool for data engineering and has been adopted by Fortune 500 companies.[4]

Overview

Airflow uses directed acyclic graphs (DAGs) to manage workflow orchestration. Tasks and dependencies are defined in Python and then Airflow manages the scheduling and execution. DAGs can be run either on a defined schedule (e.g. hourly or daily) or based on external event triggers (e.g. a file appearing in Hive[5]). Previous DAG-based schedulers like Oozie and Azkaban tended to rely on multiple configuration files and file system trees to create a DAG, whereas in Airflow, DAGs can often be written in one Python file.[6]

Managed providers

The big 3 hyper-scalers providers offer ancillary services around the core open-source project:

  • Apache Airflow Job is a SaaS managed version of open source Apache Airflow from Microsoft Azure that integrates with Microsoft Fabric for Fabric item orchestration.
  • Cloud Composer is a managed version of Airflow that runs on Google Cloud Platform (GCP) and integrates well with other GCP services.[7]
  • Amazon Web Services offers Managed Workflows for Apache Airflow starting from November 2020.[8]

References

  1. ↑ "Apache Airflow". https://airflow.apache.org/project.html. 
  2. ↑ Beauchemin, Maxime (June 2, 2015). "Airflow: a workflow management platform". https://medium.com/airbnb-engineering/airflow-a-workflow-management-platform-46318b977fd8. 
  3. ↑ "Airflow". https://airbnb.io/projects/airflow/. 
  4. ↑ Kerner, Sean Michael (April 22, 2025). "Batch data processing is too slow for real-time AI: How open-source Apache Airflow 3.0 solves the challenge with event-driven data orchestration". https://venturebeat.com/data-infrastructure/open-source-airflow-3-0-set-to-accelerate-data-orchestration-for-enterprise-ai-inference. 
  5. ↑ Trencseni, Marton (January 16, 2016). "Airflow review". http://bytepawn.com/airflow.html. 
  6. ↑ "AirflowProposal". March 28, 2019. https://cwiki.apache.org/confluence/display/incubator/AirflowProposal. 
  7. ↑ "Google launches Cloud Composer, a new workflow automation tool for developers" (in en-US). May 2018. https://techcrunch.com/2018/05/01/google-launches-cloud-composer-a-new-workflow-automation-tool-for-developers/. 
  8. ↑ "Introducing Amazon Managed Workflows for Apache Airflow (MWAA)" (in en-US). 2020-11-24. https://aws.amazon.com/blogs/aws/introducing-amazon-managed-workflows-for-apache-airflow-mwaa/. 

Further reading

  • Harenslak, Bas; de Ruiter, Julian (2021). Data Pipelines with Apache Airflow. Manning Publications (published April 27, 2021). ISBN 9781617296901. 
  • No URL found. Please specify a URL here or add one to Wikidata.



Retrieved from "https://handwiki.org/wiki/index.php?title=Software:Apache_Airflow&oldid=4290993"