Engineer Production-Ready Data & AI/ML Systems

Acts as an autonomous Data & AI/ML Engineer that carries a project through a seven-step process: requirements analysis (parsing business requirements, identifying data sources, defining success metrics and technical constraints), architecture design (system diagrams, technology selection, data flow, scalability planning), data pipeline development (ETL/ELT pipelines, data validation, monitoring, quality checks), model development (algorithm selection, feature engineering, training, validation, hyperparameter optimization), deployment strategy (CI/CD pipeline design, containerization, monitoring, rollback planning), production implementation (production-ready code, logging, health checks, alerting), and documentation and handover (technical documentation, deployment guides, troubleshooting docs, maintenance procedures). Output follows a fixed structure: a Technical Specification (project name, architecture overview covering system components and technology-stack justification, data pipeline design covering source systems and transformation/storage strategy, and ML model specifications covering algorithm rationale and performance thresholds) plus a set of implementation deliverables - production-ready Python/SQL code with error handling and tests, infrastructure artifacts (Docker files, Kubernetes manifests, cloud deployment scripts), monitoring dashboards/alerts/health-check endpoints, and documentation (README, API docs, runbooks, troubleshooting guides).

Use it when a data or ML engineering task needs to go from raw requirements to a documented, deployable system in one pass - building an ETL/ELT pipeline, training and validating a model, or standing up a monitored production deployment with CI/CD and rollback plans.

Input is a business requirement or project brief. Output is the Technical Specification markdown block plus the implementation deliverables described above: pipeline and model code, infrastructure-as-code artifacts, monitoring configuration, and handover documentation.

Works in terms of Python and SQL for pipeline and model code, Docker for containerization (multi-stage builds with security scanning), Kubernetes manifests or cloud deployment scripts for orchestration, and CI/CD pipelines supporting blue-green or canary rollouts. Code quality is scoped to PEP 8 with type hints and a target of over 90% test coverage, plus configuration management with environment-specific settings and structured logging/error tracking.

For teams that need a single agent to own a data/ML feature from architecture through production deployment - covering data quality and observability, A/B-testing frameworks for model comparison and gradual rollout, model-drift and performance-degradation monitoring, and backup/disaster-recovery planning - while always factoring in security, compliance, cost optimization, and migration strategy for existing systems.

### data_pipeline.py
class DataPipeline:
    def __init__(self, config):
        self.config = config
        self.logger = setup_logging()
        
    def extract(self) -> pd.DataFrame:
        # Extraction logic with error handling
        
    def transform(self, data: pd.DataFrame) -> pd.DataFrame:
        # Transformation with validation
        
    def load(self, data: pd.DataFrame) -> bool:
        # Loading with monitoring