Build Real-Time ML Streaming Pipelines

This skill designs streaming machine learning pipelines processing real-time data with Kafka, Spark, and Flink. It applies Kappa architecture (stream-only, replayable logs) over Lambda for most ML use cases, event-driven design with idempotent exactly-once processing, and implements: sliding-window feature aggregation tracking event count/average value/unique categories per entity with automatic window cleanup, a Redis-backed streaming feature store with TTL expiration and batch retrieval, an A/B testing model router using deterministic hash-based traffic splitting, an online learning pipeline using SGDRegressor.partial_fit on a Kafka training-data stream, Avro schema evolution handling with graceful fallback for missing fields, and statistical drift detection using the Kolmogorov-Smirnov two-sample test comparing reference and current data windows. It also covers event-driven design fundamentals: modeling around immutable events rather than mutable state, using event sourcing for model-training-data lineage, and separating prediction (command) responsibilities from model-update (query) responsibilities.

Use this skill when building a real-time ML pipeline - computing sliding-window features per entity, serving features from a low-latency store like Redis, routing prediction traffic across model versions for A/B testing, incrementally updating a model with online learning, handling schema evolution in a Kafka/Avro pipeline, or detecting statistical drift between reference and live data distributions. It also applies once that pipeline is running in production: managing backpressure with circuit breakers for downstream services and adaptive batching based on throughput, and scaling it - horizontal scaling with consistent hashing for stateful operations, separating compute from storage scaling, and auto-scaling on queue depth and CPU metrics with graceful shutdown for zero-downtime deploys.

It does not cover batch ML training pipelines or one-off model training scripts - it is focused specifically on real-time, continuously-updating streaming ML architecture.

Inputs are typically a stream of events (Kafka topics) and a model to serve or train. Outputs include feature engineering code, for example sliding-window feature computation:

class SlidingWindowFeatures:
    def get_features(self, user_id):
        events = list(self.windows[user_id])
        return {
            'event_count': len(events),
            'avg_value': sum(e[1].get('value', 0) for e in events) / len(events),
            'unique_categories': len(set(e[1].get('category') for e in events))
        }

Other outputs include a Redis-backed feature store with TTL and batch mget retrieval, a hash-based A/B model router, an online learning pipeline using partial_fit on streamed training data, an Avro schema evolution handler with fallback defaults, and a Kolmogorov-Smirnov drift detector comparing reference vs. current windows. It also produces guidance for testing the whole pipeline - a Docker Compose setup wiring Kafka, Redis, and the ML pipeline service together for integration testing - plus model-validation practices: shadow-mode testing for new models before they take live traffic, statistical tests for ongoing prediction-quality monitoring, alerting on model performance degradation, and holdout datasets kept for continuous validation.

ML and data engineers building real-time prediction or online-learning systems who need streaming-specific feature engineering, model serving, drift detection, and the operational practices (backpressure handling, scaling, integration testing, shadow-mode validation) to run that pipeline reliably in production, rather than batch-oriented ML tooling.