Automate ML CI/CD Pipelines

Designs, implements, and maintains CI/CD pipelines for machine learning systems, covering MLOps practices, model versioning, automated testing, deployment patterns, and the operational challenges specific to productionizing ML. Architecture follows five principles: multi-stage validation (data validation, training, testing, and deployment as distinct stages), artifact management (versioning datasets, models, metrics, and configurations), environment consistency across dev/staging/production, safe rollback mechanisms for model deployments, and observability built into every stage - organized into four pipeline components: a data pipeline (ingestion, validation, preprocessing, feature engineering), a training pipeline (training, hyperparameter tuning, evaluation), a deployment pipeline (packaging, testing, production rollout), and a monitoring pipeline (performance tracking, drift detection, alerting). It implements this as a concrete GitHub Actions workflow with four sequential jobs (data validation, model training with artifact upload, model testing across unit/integration/performance suites, and staging deployment gated on the main branch), a ModelValidator class that checks current accuracy/precision/recall/F1 against baseline metrics within a tolerance threshold and runs Kolmogorov-Smirnov tests per numeric column to detect data drift, a Kubernetes deployment with liveness/readiness probes and resource limits behind a LoadBalancer service, and a BlueGreenDeployment class that deploys to the inactive color, runs a smoke test against a /predict endpoint, and either switches traffic or rolls back automatically on validation failure. Monitoring combines Prometheus metrics (a prediction counter, a latency histogram, an accuracy gauge) with MLflow experiment logging of model version, predictions, and accuracy.

Use it to build or harden an ML deployment pipeline that needs more than a generic CI/CD template - baseline-metric validation before promoting a model, statistical data-drift detection, blue-green rollout with an automated rollback path, and production monitoring wired to both Prometheus and MLflow.

Input is a model training/evaluation codebase and its target deployment environment. Output is a GitHub Actions pipeline definition, a Dockerfile for serving the model (non-root user, health check, port 8000), Kubernetes Deployment and Service manifests, a ModelValidator for performance and drift checks, a BlueGreenDeployment script for safe rollout, and monitoring instrumentation combining Prometheus metrics with MLflow logging.

Built on GitHub Actions for CI/CD orchestration, Docker and Kubernetes for packaging and serving, scikit-learn and scipy (ks_2samp) for model and drift validation, Prometheus (Counter, Histogram, Gauge) for real-time metrics, and MLflow for experiment tracking and model logging.

For ML engineers and MLOps teams productionizing model deployments rather than running training as a one-off script. It also covers pipeline security (secret management for credentials, access controls on pipeline endpoints, container vulnerability scanning, least-privilege service accounts), performance optimization (caching feature computation and model artifacts, parallel batch processing, resource-aware autoscaling), a five-part testing strategy (unit, integration, performance, data-validation, and model-validation tests), and troubleshooting guidance for failed deployments, resource constraints, data-pipeline failures, and model performance degradation.

FROM python:3.9-slim

### Set working directory
WORKDIR /app

### Install system dependencies
RUN apt-get update && apt-get install -y \
    gcc \
    && rm -rf /var/lib/apt/lists/*

### Copy requirements and install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

### Copy application code
COPY src/ ./src/
COPY models/ ./models/
COPY config/ ./config/

### Create non-root user
RUN useradd --create-home --shell /bin/bash ml-user
USER ml-user

### Health check
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
    CMD python src/health_check.py

### Expose port
EXPOSE 8000

### Run application
CMD ["python", "src/serve.py"]