Implement ML A/B Testing Framework

This skill equips an agent with expertise in designing, implementing, and analyzing A/B tests specifically for machine learning systems in production - covering concept drift, model bias, statistical power calculations, and the challenge of measuring both business and model-performance metrics together. It walks through pre-registering primary/secondary metrics and computing minimum detectable effect (MDE) and required sample size before a test starts, correcting for multiple testing when evaluating several metrics, choosing the right randomization unit (user, session, or request), monitoring both model metrics (accuracy, AUC, precision/recall) and business metrics (conversion, revenue, engagement) side by side, and accounting for inference latency, compute cost, seasonality, and model versioning/reproducibility across the experiment.

It provides worked Python examples for sample-size calculation via statsmodels' ttest_power:

import numpy as np
from scipy import stats
from statsmodels.stats.power import ttest_power

def calculate_sample_size(baseline_rate, mde, alpha=0.05, power=0.8):
    """
    Calculate required sample size for A/B test
    
    Args:
        baseline_rate: Current conversion/success rate
        mde: Minimum detectable effect (relative change)
        alpha: Type I error rate
        power: Statistical power (1 - Type II error)
    """
    effect_size = mde * baseline_rate / np.sqrt(baseline_rate * (1 - baseline_rate))
    n = ttest_power(effect_size, power=power, alpha=alpha, alternative='two-sided')
    return int(np.ceil(n))

Other worked examples cover consistent-hashing traffic splitting (ABTestSplitter, MD5-hash bucketing into control/treatment), a model-serving wrapper (ABTestModelServer) that logs prediction, variant, latency, and timestamp per request, Bayesian A/B analysis with pymc3 (Beta priors, binomial likelihood, posterior lift and probability-of-positive-lift), a SequentialABTest class implementing early stopping with an O'Brien-Fleming alpha-spending function, and ModelDriftMonitor, which flags prediction drift via a Kolmogorov-Smirnov test and Jensen-Shannon divergence against a baseline.

Use it when designing or reviewing an A/B test that involves a deployed ML model - anywhere concept drift, inference latency, or model versioning could confound a standard web A/B test. It is not a general-purpose A/B testing primer for simple UI/copy tests with no model in the loop, and it does not replace a dedicated experimentation platform - it supplies the statistical design and monitoring logic, not the traffic-routing infrastructure itself.

Given an experiment brief (baseline rate, desired MDE, or a description of the model change being tested), it returns a test design: required sample size, traffic-split logic, metric set (model plus business), a statistical analysis approach (frequentist or Bayesian), and a drift-monitoring check. Code samples are ready-to-adapt Python built on scipy, statsmodels, pymc3, and arviz.

The code samples integrate with numpy, scipy.stats, statsmodels.stats.power, pymc3/arviz for Bayesian inference, and scipy.spatial.distance.jensenshannon plus scipy.stats.ks_2samp for drift detection - standard Python data-science libraries, not a Spark/VibeBaza-specific SDK.

ML engineers and data scientists running production experiments on deployed models, who need the experiment design to account for model-specific effects rather than treating the model as a black box.