Orchestrate ML Pipelines with Kubeflow

This skill builds ML workflow pipelines on Kubeflow, the machine-learning orchestration platform built on Kubernetes. Core architecture principles cover containerized, isolated components per pipeline step, typed input/output contracts between components, Kubeflow's artifact store for intermediate data, and per-component CPU/memory/GPU resource specification. It covers compiling pipelines with KFP SDK v2 for improved type safety, compiling to YAML for version control and deployment, and implementing conditional and parallel execution patterns.

It provides working component patterns: lightweight Python function components decorated with a base image and installed packages, and container-based components for more complex dependencies using a custom image. Pipeline orchestration covers a full conditional/parallel example - ingesting data, preprocessing splits in parallel via ParallelFor, training a model, and gating deployment behind a model-quality condition that only proceeds if accuracy clears a threshold and deployment is explicitly requested. Resource management covers GPU limits, memory and CPU limits, node-selector constraints for specific hardware, and tolerations for tainted nodes. Advanced configuration covers pipeline caching for expensive operations, custom artifact storage locations, cleanup via exit handlers, and multi-tenant security (service-account-based cloud secret access, non-root security contexts).

Deployment guidance covers versioned pipeline compilation, programmatic deployment and execution via the KFP client, and structured logging with custom pipeline-run-ID metrics for observability. Performance and troubleshooting guidance covers avoiding over-provisioned resource requests, streaming large datasets via volume mounts, caching for iterative development, spot instances with appropriate tolerations for cost savings, structured debug logging, init containers for dependency setup, exponential-backoff retry policies, inter-stage validation, and pipeline metadata/lineage tracking for reproducibility.

train_task = gpu_training_component(
    dataset=preprocess_task.outputs['data']
).set_gpu_limit(1).set_memory_limit('16Gi').set_cpu_limit('4')

Use this skill when building or deploying an ML pipeline on Kubeflow - designing components, implementing conditional or parallel orchestration, configuring GPU/resource requirements, or setting up versioned CI/CD deployment and monitoring.

It is not a fit for choosing Kubeflow over alternative ML orchestration platforms, or for the model training logic itself beyond how it's wrapped as a pipeline component - it's scoped to pipeline architecture, component development, and deployment on Kubeflow specifically.

Inputs are the ML workflow steps you need orchestrated (data ingestion, preprocessing, training, evaluation, deployment) and their resource requirements. Outputs are working KFP SDK v2 component and pipeline code, a compiled versioned YAML pipeline, resource and security configuration, and deployment/monitoring code for production runs.

ML engineers building production ML pipelines on Kubeflow who need concrete, working patterns for component development, conditional/parallel orchestration, GPU resource management, and CI/CD deployment - rather than assembling pipeline architecture from the KFP SDK documentation alone.