Orchestrate Data Pipelines with Airflow Patterns

This is a skill providing production-ready patterns for Apache Airflow, covering DAG design, operators, sensors, testing, and deployment strategies for data pipeline orchestration. The skill itself points to a companion resource file, resources/implementation-playbook.md, which holds the detailed patterns, checklists, and templates; the top-level instructions set the workflow to follow: identify data sources, schedules, and dependencies; design idempotent tasks with clear ownership and retries; implement DAGs with observability and alerting hooks; and validate in staging with documented operational runbooks before going live.

Use this skill when creating data-pipeline orchestration with Airflow - designing DAG structures and dependencies, implementing custom operators and sensors, testing DAGs locally, setting up Airflow in production, or debugging failed DAG runs. Don't use it if you only need a simple cron job or shell script, if Airflow isn't part of your tooling stack, or if the task has nothing to do with workflow orchestration. It also carries an explicit safety note: don't change production DAG schedules without approval, and test backfills and retries carefully to avoid duplicating data.

Input is a data pipeline's sources, schedules, and dependencies; output is an idempotent DAG design with retries, observability, and alerting built in, validated in staging with a documented operational runbook before production rollout. The detailed implementation patterns, checklists, and templates live in the companion resources/implementation-playbook.md file rather than in the top-level instructions themselves.

Data engineers building or operating Apache Airflow pipelines who need a checklist for safe DAG design, testing, and production deployment, particularly around idempotency, backfill safety, and change approval for production schedules.