Detect Anomalies with Statistical and ML Rules

This skill provides expertise in designing and implementing anomaly detection rules, with deep knowledge of statistical methods, machine learning approaches, and real-time monitoring systems, aiming to minimize false positives while maintaining high sensitivity to real anomalies. Core principles include a statistical foundation (establishing a baseline of normal behavior via mean/median/percentiles/standard deviation, analyzing data distributions to pick the right method, accounting for seasonality and trends in time-series data, and considering correlations across multiple metrics) and a rule design philosophy (specificity over raw sensitivity to reduce false positives while keeping reasonable detection rates, business-context-aware rules, adaptive thresholds that adjust to changing baselines, and confidence scores rather than binary classification).

Statistical detection methods include Z-score detection (rolling mean/std with an adaptive baseline, flagging values whose Z-score exceeds a threshold) and the IQR method (rolling quartiles defining lower/upper bounds, robust to outliers). Time-series rules cover seasonal decomposition anomaly detection (removing trend/seasonal components via seasonal_decompose, then applying Isolation Forest to the residuals) and rate-of-change detection (flagging values whose percentage or absolute change exceeds a threshold). Cross-metric correlation rules detect anomalies when two normally-correlated metrics deviate from their expected linear relationship beyond a deviation threshold. Real-time streaming rules use an EMA-based anomaly detector that maintains an exponential moving average and variance estimate incrementally, computing a live anomaly score and flag on each new value without needing a full window in memory.

Configuration templates express a multi-rule system in YAML - per-metric rule stacks (threshold, Z-score, rate-of-change for one metric; IQR and seasonal for another), cross-metric correlation rules, and global settings (aggregation method, minimum confidence, cooldown period to prevent alert storms). Best practices cover rule optimization (backtesting against historical data with known anomalies, A/B testing rule performance across time periods, ROC/precision-recall analysis for threshold tuning, and ensemble methods combining multiple detectors), production considerations (computational efficiency for real-time processing, memory management via rolling windows and incremental statistics, smart alerting with suppression and escalation, and monitoring for rule/model drift with periodic retraining), and false-positive reduction (incorporating known maintenance windows and expected events, requiring multi-rule confirmation before alerting, using confidence intervals rather than hard thresholds, and feedback loops that learn from false-positive patterns).

Use this skill when designing anomaly detection rules for metrics, time series, or correlated signals that need statistically grounded, low-false-positive alerting - infrastructure metrics, business KPIs, or streaming data. It is well suited to systems with historical data available for backtesting and threshold tuning. It is not meant for a single hard-coded threshold check with no statistical basis, or for data with too little history to establish a meaningful baseline.

Input: the metric or time series to monitor, its historical data, and known correlated metrics.

Output: anomaly detection logic (Z-score, IQR, seasonal, rate-of-change, correlation, or streaming EMA-based) and a rule configuration. Example rolling Z-score detector:

def zscore_anomaly_detection(data, threshold=3, window=30):
    rolling_mean = data.rolling(window=window, min_periods=10).mean()
    rolling_std = data.rolling(window=window, min_periods=10).std()
    z_scores = np.abs((data - rolling_mean) / rolling_std)
    return {'anomalies': z_scores > threshold, 'scores': z_scores}

Works with Python's pandas/numpy/scipy/statsmodels/scikit-learn stack for statistical analysis and Isolation Forest modeling, and expresses multi-rule configurations in YAML for monitoring systems.

Data and platform engineers building anomaly detection for infrastructure or business metrics, and teams that need low-false-positive alerting backed by statistical rigor rather than a single hard threshold.