Detect Anomalies with Statistical and ML Rules

Anomaly Detection Rule Expert Agent designs and implements rules for data engineering and monitoring systems. It applies statistical and machine learning methods to time series and multiple metrics, aiming to reduce false positives while maintaining sensitivity to genuine anomalies. Its core principles include establishing baselines, understanding distributions, accounting for seasonality and trends, incorporating business context, adapting thresholds, and returning confidence scores instead of only binary labels.

Guidance covers rolling Z-scores, interquartile range bounds, seasonal decomposition with Isolation Forest, percentage rate-of-change checks, correlated metric residuals, and streaming detection with an exponential moving average. A YAML template combines threshold, z-score, rate-change, IQR, seasonal, and correlation rules, with weighted aggregation, minimum confidence, and cooldown settings.

Use this agent when monitoring systems need anomaly rules for changing baselines, seasonality, correlated metrics, or streaming values. It can help choose statistical methods, configure thresholds, score anomalies, and combine rule outputs. It is useful when a simple static threshold creates too many false positives or misses context-dependent changes.

Rules need validation against the domain and historical data. Use backtesting against known anomalies, A/B testing rules across time periods, tuning thresholds with ROC and precision-recall analysis, and combining detection methods. Account for maintenance windows and expected events, use multi-stage confirmation where suitable, consider confidence intervals, and learn from false-positive feedback. Monitor drift and periodically retrain where applicable.

Inputs include historical or streaming metrics, baseline distributions, time series frequency, seasonal period, correlated metric pairs, known events, and acceptable false-positive and detection rates. The agent can propose rule thresholds and windows, generate implementation code, configure aggregation, or design alert behavior. It can return anomaly labels, scores, baselines, bounds, residuals, and related trend or seasonal components depending on the method.

Examples use a rolling window for Z-score and IQR detection, seasonal decomposition before Isolation Forest, rate-of-change thresholds, linear regression residuals for correlated metrics, and an EMA detector that tracks a variance estimate. A multi-rule configuration includes upper and lower bounds, thresholds, confidence, and a cooldown period. These are examples, not universal operating values.

The examples use Python, NumPy, pandas, SciPy, statsmodels, and scikit-learn. Methods include rolling statistics, seasonal_decompose, IsolationForest, linear regression, and streaming EMA updates. The agent is framed for real-time monitoring and data engineering contexts.

This agent is for data engineers, ML practitioners, and monitoring teams that need explainable anomaly rules across univariate, multivariate, and streaming data. It fits teams that can provide historical behavior and domain context and can measure rule quality over time.

from scipy import stats