Build Production-Ready Airflow DAGs

This skill provides expertise in Apache Airflow DAG development, specializing in reliable, scalable, maintainable workflow orchestration - Airflow architecture, the TaskFlow API, XComs, sensors, operators, and advanced scheduling patterns. Core DAG design principles include idempotency (each task produces the same result on repeated runs), atomicity (tasks are self-contained and fail fast), backfill-friendliness (DAGs correctly handle historical data), observability (comprehensive logging and monitoring), and resource efficiency (appropriate pools, queues, and resource limits).

DAG structure best practices set default_args (owner, start_date, retries, retry_delay, execution_timeout, email-on-failure) and DAG-level settings (schedule_interval, catchup, max_active_runs, tags) for production-ready pipelines. TaskFlow API patterns use the modern @task decorator for Python tasks with automatic XCom handling, chaining extract/transform/load functions with clear data dependencies. Advanced scheduling and dependencies cover file sensors with timeout and poke_interval, database operators for SQL-based checks (e.g. PostgresOperator with a HAVING clause enforcing a minimum row-count threshold), and conditional email notifications with trigger rules like all_success.

Error handling and data quality covers custom validation tasks that raise on insufficient data or quality thresholds and attach quality metrics to the task's output, plus @task.branch for volume-based conditional execution paths. Configuration management uses Airflow Variables for tunable parameters (batch size, source endpoints, notification emails) and BaseHook connections for secure credential retrieval. Monitoring and observability covers structured pipeline metrics logging (execution date, duration, records processed, success rate) and Slack webhook alerts on failure with a one_failed trigger rule. Performance optimization tips include using the pool parameter to limit concurrent resource use, tuning max_active_tasks/max_active_runs, dynamic task generation for parallelization, appropriate sensor poke_interval/timeout values, task groups for complex workflows, per-task execution_timeout, depends_on_past=False unless truly needed, and controlled logging levels to avoid log spam. Testing strategies include DAG integrity tests (confirming a DAG imports without errors and has tasks) and task dependency tests (verifying the expected upstream/downstream structure) using Airflow's DagBag.

Use this skill when building or hardening an Apache Airflow DAG that needs idempotent, backfill-safe execution, proper error handling, data quality gates, and monitoring/alerting. It is well suited to production ETL/ELT pipelines with real reliability requirements. It is not meant for simple one-off scripts with no scheduling or orchestration need, or for orchestration platforms other than Airflow.

Input: the data pipeline's extract/transform/load steps, schedule requirements, and data quality thresholds.

Output: a complete, production-ready Airflow DAG with TaskFlow tasks, quality checks, monitoring, and tests. Example TaskFlow pattern:

@task(retries=3, retry_delay=timedelta(minutes=2))
def extract_data(ds: str, **context) -> dict:
    """Extract data with date partitioning"""
    data = {'records_count': 1000, 'extraction_date': ds}
    return data

@task
def transform_data(raw_data: dict) -> dict:
    return {'processed_records': raw_data['records_count'] * 0.95}

Builds on Apache Airflow's operators and providers (PostgresOperator, S3KeySensor, SlackWebhookOperator, EmailOperator), Airflow Variables and connections for configuration, and pytest with DagBag for testing.

Data engineers building or maintaining production Airflow pipelines who need idempotent, well-tested DAGs with data quality gates, and teams that want proper monitoring and alerting built into their orchestration from the start.