Orchestrate Data Pipelines with Airflow Patterns

This skill provides production-ready patterns for building Apache Airflow DAGs, covering DAG design principles, operators, sensors, testing, and deployment strategies. It packages six patterns: the TaskFlow API (Airflow 2.0+) for defining tasks with the @task decorator and automatic XCom passing between extract/transform/load steps; dynamic DAG generation from a list of pipeline configurations via a factory function that instantiates one DAG per entry; branching and conditional logic using BranchPythonOperator with a quality-score based routing example and a join step using TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS; sensors for external dependencies, including S3KeySensor for waiting on an S3 object, ExternalTaskSensor for waiting on another DAG, and a custom @task.sensor-decorated API-availability check; error handling with a task_failure_callback and dag_failure_callback pair plus a cleanup task using TriggerRule.ALL_DONE; and DAG testing with DagBag, checking for import errors, task structure, dependency wiring, and cycle detection via pytest. It follows four core design principles: idempotent tasks (running twice produces the same result), atomic tasks (succeed or fail completely), incremental processing (only new/changed data), and observability (logs, metrics, and alerts at every step).

Use this skill when creating data pipeline orchestration with Airflow, designing DAG structures and dependencies, implementing custom operators and sensors, testing Airflow DAGs locally, setting up Airflow in production, or debugging failed DAG runs. Do not use it when a simple cron job or shell script is enough, when Airflow is not part of the tooling stack, or when the task is unrelated to workflow orchestration.

Inputs are the pipeline's data sources, schedules, and dependencies, identified before DAG design begins; config-driven examples take a list of pipeline configurations (name, schedule, source) and generate one DAG per entry via a factory function. Outputs are working DAG definitions validated in staging plus documented operational runbooks; the detailed patterns, checklists, and code samples live in a companion resources/implementation-playbook.md referenced from the skill's top-level instructions, alongside a recommended project structure separating dags/, plugins/, and tests/ directories.

Examples reference Amazon S3 through S3KeySensor and the airflow.providers.amazon.aws.sensors.s3 provider, pandas for reading CSV from S3 and writing Parquet, and a generic REST API polled through a custom sensor. DAGs can wait on the completion of other DAGs via ExternalTaskSensor, and failure callbacks are stubbed out for routing alerts to a Slack-style notification channel.

The skill also names two safety guardrails - avoid changing production DAG schedules without approval, and test backfills and retries carefully to prevent data duplication - and a set of do's and don'ts: use the TaskFlow API, set timeouts to prevent zombie tasks, and use mode='reschedule' for sensors to free up worker slots; never use depends_on_past=True (creates bottlenecks), hardcode dates instead of {{ ds }} macros, or put heavy logic directly in the DAG file.

Data engineers and platform teams building or maintaining Airflow-based orchestration who need vetted patterns for DAG design, testing, and production deployment rather than starting from a blank DAG file.