Orchestrate End-to-End ML Pipelines

A comprehensive MLOps pipeline skill covering the full lifecycle: data ingestion, preparation, training, validation, deployment, and monitoring. Core capabilities span five areas: pipeline architecture (DAG orchestration patterns for Airflow, Dagster, or Kubeflow, component dependencies, retry strategies), data preparation (validation and quality checks, feature engineering, data versioning and lineage, train/validation/test splitting), model training (job orchestration, hyperparameter management, experiment tracking, distributed training), model validation (validation frameworks, A/B testing infrastructure, performance regression detection, model comparison), and deployment automation (model serving, canary deployments, blue-green strategies, rollback mechanisms). It ships reference guides (data-preparation.md, model-training.md, model-validation.md, model-deployment.md) and templates (a pipeline-dag.yaml.template, a training-config.yaml, a validation-checklist.md). The documented production workflow moves through four phases - data preparation (ingest, quality-check, transform, version), training (load versioned data, execute jobs, track experiments, save models), validation (run the test suite, compare against baseline, generate reports, approve), and deployment (package artifacts, deploy to serving infrastructure, configure monitoring, validate production traffic). A progressive-disclosure ladder scales pipeline complexity across five levels, from a simple linear data-to-train-to-deploy flow up to multi-model pipelines with ensemble strategies, adding validation and monitoring, hyperparameter tuning, and A/B testing with gradual rollout at each step in between.

Use when building a new ML pipeline from scratch, designing DAG-based orchestration for ML systems, automating the data-to-model-to-deployment chain, setting up reproducible training workflows, or integrating ML components into a production system. Not for tasks unrelated to ML pipeline orchestration; for deeper work on adjacent concerns it points to sibling skills - experiment-tracking-setup for MLflow/W&B, hyperparameter-tuning for optimization, and model-deployment-patterns for advanced serving strategies - rather than duplicating that scope here.

Input is raw data sources plus pipeline requirements (methodology, orchestration tool, deployment target). Output is a DAG-based pipeline definition (stages with explicit dependencies, following the pipeline-dag.yaml.template shape), trained and versioned model artifacts, a validation report comparing against baseline, and a deployed, monitored serving endpoint with rollback capability.

stages:
  - name: data_preparation
    dependencies: []
  - name: model_training
    dependencies: [data_preparation]
  - name: model_evaluation
    dependencies: [model_training]
  - name: model_deployment
    dependencies: [model_evaluation]

Names Apache Airflow, Dagster, Kubeflow Pipelines, and Prefect for orchestration; MLflow and Weights & Biases for experiment tracking and model registry, plus TensorBoard for training metrics; AWS SageMaker, Google Vertex AI, Azure ML, and Kubernetes with KServe for deployment; and Great Expectations, TFX, or DVC for data validation and versioning.

ML engineers and platform teams building or hardening a production ML pipeline who need a concrete phase structure, tool choices, and troubleshooting steps for each stage rather than assembling an MLOps stack from first principles.