Open Source Project #177: Apache Airflow — Workflow Orchestration in Python, the Standard Tool for Data Engineers

Apache Airflow is a platform for programmatically authoring, scheduling, and monitoring workflows. Core concept: DAGs (Directed Acyclic Graphs) — define task dependencies in Python code, trigger by schedule or data events, monitor every step in the Web UI. Covers four major scenarios: ETL/ELT pipelines, ML training pipelines, scheduled reporting, and infrastructure automation. 46.4k Stars, Apache 2.0, version 3.3.0, 600+ built-in Operators and Providers.

·10 min read·Data Engineering
open-sourceworkfloworchestrationdata-engineeringETLMLOpsPythonDAGApacheschedulerdata-pipeline

Introduction

"If your code needs to run in order, on a schedule, with retry logic on failure — you need an orchestrator."

This is article #177 in the "One Open Source Project a Day" series. Today's project is Apache Airflow — a platform for writing and scheduling workflows in Python, one of the most widely used open-source tools in data engineering.

Before diving into the technology, the most important question: what can Airflow actually do?

Short answer: any work that involves "executing multiple steps in order, with dependencies between steps, triggered on a schedule or by a condition" can be managed with Airflow. The classic use case is a data pipeline — pull data from a database at 2am, clean it, write it to the data warehouse, generate a report. But it goes beyond that: training machine learning models, sending email reports, automating infrastructure operations all have real production deployments on Airflow.

46,400 Stars. Apache 2.0. Version 3.3.0. Used in production by thousands of companies worldwide.

What You'll Learn

  • The four core problem categories Airflow solves (the important part)
  • What a DAG is and why workflows map naturally to graphs
  • Operators and Providers: what the 600+ integrations cover
  • Airflow 3.0's key new features: event-driven scheduling and Asset-aware workflows
  • Quick start: a real DAG in 10 lines of Python
  • When to use Airflow — and when not to

Prerequisites

  • Basic Python familiarity
  • Understanding what a scheduled job is (cron, etc.)
  • General awareness of data processing (no need to be a professional data engineer)

What Airflow Does (The Core Question)

This is the most important section.

Use Case 1: ETL/ELT Data Pipelines

The primary use case. The typical problem data engineers face every day:

Every day at 2am:
  1. Pull yesterday's order data from the MySQL production database
  2. Clean and transform (dedup, format standardization, enrich with dimension data)
  3. Write to BigQuery / Snowflake data warehouse
  4. Refresh the BI dashboard
  5. If step 3 fails, send a Slack alert and retry
 
These 5 steps have a strict execution order.
Step 4 can only start after step 3 succeeds.

Airflow models this as a DAG, configures the dependency relationships, triggers it automatically every night, logs every step's result, handles retries, and shows every historical run in the Web UI.

Real scale: Airflow was created at Airbnb to manage hundreds of daily ETL tasks. Today, some companies run thousands of DAGs in production with hundreds of thousands of task executions per day.

Use Case 2: ML Training Pipelines

MLOps workflows follow a recognizable pattern:

Every Monday:
  1. Pull latest training data from the data warehouse
  2. Feature engineering (normalize, encode, split train/test)
  3. Train the model (Python script or submit to a Spark cluster)
  4. Evaluate model metrics (precision, recall, AUC)
  5. If metrics exceed threshold → automatically deploy to production
  6. If below threshold → notify the data science team

Steps 5 and 6 are conditional branches — Airflow's BranchPythonOperator handles routing based on the previous step's result.

Use Case 3: Scheduled Reporting and Data Sync

Not every scenario involves complex big data:

  • Send yesterday's sales data to management every morning at 9am (email + Excel attachment)
  • Sync new CRM customers to the marketing platform every hour
  • Consolidate department KPIs into Google Sheets every Friday
  • Generate the previous month's financial report and upload to S3 on the 1st of each month

These tasks used to be cron + shell scripts. Airflow adds observability: did each run succeed, which step was slow, what happened when it failed — all visible in one place.

Use Case 4: Infrastructure Automation

Airflow isn't limited to data scenarios:

  • Daily detection and archival of S3 files older than 30 days
  • Scheduled database backups with automated backup verification
  • Automated cloud resource scaling (scale up at peak, scale down at off-hours)
  • Integration test orchestration in CI/CD pipelines

Core Concept: The DAG

Why Graphs for Workflows

DAG = Directed Acyclic Graph. Each node is a task; edges represent "must finish A before starting B." "Acyclic" prevents deadlocks (A waiting on B waiting on A).

A typical DAG (data pipeline):
 
extract_data ──→ transform_data ──→ load_to_warehouse ──→ send_report
                      ↓
              validate_schema ──→ quarantine_bad_data

Airflow defines this graph in Python:

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta
 
with DAG(
    dag_id="daily_sales_pipeline",
    schedule="0 2 * * *",       # every day at 2am
    start_date=datetime(2026, 1, 1),
    catchup=False,
    default_args={
        "retries": 2,
        "retry_delay": timedelta(minutes=5),
    },
) as dag:
 
    extract = PythonOperator(
        task_id="extract_data",
        python_callable=extract_from_mysql,
    )
 
    transform = PythonOperator(
        task_id="transform_data",
        python_callable=clean_and_transform,
    )
 
    load = PythonOperator(
        task_id="load_to_warehouse",
        python_callable=load_to_bigquery,
    )
 
    report = BashOperator(
        task_id="send_report",
        bash_command="python send_email.py --date {{ ds }}",
    )
 
    # Define execution order
    extract >> transform >> load >> report

These 40 lines of Python are a complete production data pipeline. Airflow handles triggering it at 2am, executing tasks in order, retrying on failure, and displaying every run's status in the Web UI.

Scheduling Options

# Cron-based scheduling
schedule="0 9 * * 1-5"   # weekdays at 9am
 
# Airflow shorthand
schedule="@daily"
schedule="@hourly"
schedule="@weekly"
 
# New in 3.0: Asset event trigger (data-driven scheduling)
from airflow.sdk import Asset
schedule=Asset("s3://my-bucket/raw-data/")  # trigger when this dataset updates

Parameterized and Dynamic DAGs

Because DAGs are Python code, you can use Python to generate them dynamically:

# Generate the same task structure for 10 regions with a loop
for region in ["us-east", "eu-west", "ap-south", ...]:
    PythonOperator(
        task_id=f"process_{region}",
        python_callable=process_region,
        op_kwargs={"region": region},
    )

Operators and Providers: 600+ Built-In Integrations

Operators are Airflow's "task templates" — each one encapsulates interaction logic with a specific system.

Built-In Operators (No Extra Install)

OperatorPurpose
PythonOperatorExecute any Python function
BashOperatorExecute shell commands
BranchPythonOperatorRoute to different branches based on a condition
EmailOperatorSend email
HttpOperatorCall HTTP APIs
TriggerDagRunOperatorTrigger another DAG
ShortCircuitOperatorSkip downstream tasks when condition isn't met

Provider Packages (Install as Needed)

Providers are Operator collections for specific platforms, installed with pip install apache-airflow-providers-XXX:

Cloud services

  • aws — S3, Redshift, EMR, Lambda, Glue, SageMaker
  • google — BigQuery, GCS, Dataflow, Vertex AI, Pub/Sub
  • azure — Blob Storage, Data Lake, Synapse, Azure ML

Databases

  • postgres, mysql, snowflake, databricks, spark

Message queues

  • apache-kafka, rabbitmq, redis

Other tools

  • slack, github, http, ssh, docker, kubernetes

Reading from S3 and writing to BigQuery in practice:

from airflow.providers.amazon.aws.transfers.s3_to_gcs import S3ToGCSOperator
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
 
s3_to_gcs = S3ToGCSOperator(
    task_id="s3_to_gcs",
    bucket="my-s3-bucket",
    prefix="data/2026-08-03/",
    dest_gcs="gs://my-gcs-bucket/",
)
 
bq_load = BigQueryInsertJobOperator(
    task_id="load_to_bq",
    configuration={
        "load": {
            "sourceUris": ["gs://my-gcs-bucket/data/*"],
            "destinationTable": {
                "projectId": "my-project",
                "datasetId": "sales",
                "tableId": "daily_orders",
            },
        }
    },
)
 
s3_to_gcs >> bq_load

Airflow 3.0: Key Changes

Airflow 3.0 launched in 2025. Two most important new features:

Event-Driven Scheduling (Asset Watchers)

Before 3.0, Airflow was primarily cron-driven — run at a fixed time, whether or not the data is ready.

3.0 introduces Assets (data assets): a DAG can "subscribe" to a data asset and trigger automatically when that asset updates, rather than waiting for a fixed time.

from airflow.sdk import Asset, DAG
 
raw_orders = Asset("s3://data-lake/raw/orders/")
 
# This DAG triggers when raw_orders is updated
with DAG(
    dag_id="process_orders",
    schedule=raw_orders,   # data-driven, not time-driven
):
    ...

Asset Watchers let you continuously monitor a message queue (Kafka, SQS, etc.) for near-real-time event-driven triggering:

from airflow.providers.standard.asset.watchers import KafkaAssetWatcher
 
my_asset = Asset(
    "orders-stream",
    watchers=[KafkaAssetWatcher(topic="new-orders", ...)],
)

DAG Versioning

3.0 adds version numbers to each DAG. When you modify DAG code, historical run records retain a snapshot of the original version; new runs use the new version. This resolves a long-standing pain point: previously, changing a DAG made historical records inconsistent.

Web UI: Visual Monitoring

Airflow's Web UI is one of its core selling points. After starting it:

Grid view: Every task's status for every DAG run, arranged left to right by time. Green = success, red = failure, yellow = running. Problems jump out immediately.

Graph view: The DAG as a directed graph, node colors reflecting current run status. Click any node to view logs, re-run a single task, or check execution time.

Assets view (new in 3.0): Dependency graph of data assets, showing which DAGs produce data and which consume it.

Quick Start

Installation (Simplest Path)

python -m venv airflow-env
source airflow-env/bin/activate
 
AIRFLOW_VERSION=3.3.0
PYTHON_VERSION=3.12
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
 
# Initialize database and start (single-machine dev mode)
airflow standalone

Open http://localhost:8080, log in with admin/admin.

Production Deployment

Use the official Helm chart for Kubernetes:

helm repo add apache-airflow https://airflow.apache.org
helm install airflow apache-airflow/airflow \
  --namespace airflow \
  --create-namespace

Or use managed services: Astronomer (commercial hosting), Amazon MWAA, Google Cloud Composer.

When to Use Airflow — and When Not To

Good fit

  • Batch workflows: steps with clear dependencies, triggered on schedule or by events
  • Observability requirements: track which runs failed, which steps were slow, query historical runs
  • Multi-system integration: data moving through MySQL → Spark → S3 → Snowflake
  • Team collaboration: multiple data engineers jointly maintaining many pipelines

Poor fit

  • Stream processing: millisecond-latency real-time processing needs Kafka Streams or Flink
  • Simple cron jobs: if you just need one script on a schedule, crontab is enough
  • API services: Airflow is a scheduler, not a web framework
  • Sub-minute triggers: Airflow's scheduling granularity is minute-level; second-level triggers need something else

Resources

Summary

Airflow's value in one sentence: it turns "a series of interdependent tasks" from hand-maintained scripts into engineered workflows — scheduled, automatically retried on failure, with queryable history, maintainable by a team.

Its core value isn't "executing tasks" — it's "managing relationships between tasks." Who runs first, who depends on whom, what happens on failure, who gets notified on success, what triggers the whole thing.

Airflow 3.0 extends the trigger model from "time-based" to "data-ready," shifting pipelines from time-driven to event-driven. For data warehouse scenarios, this matters: instead of waiting at 2am hoping upstream data is ready, the downstream pipeline starts the moment upstream data arrives.

46,000 Stars, 10 years of production validation, thousands of companies depending on it. If your work involves data pipelines, scheduled tasks, or multi-step automation, Airflow is the most mature community choice available.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.

← Back to Blog