Automate Experiment Tracking and Analysis

This agent designs, implements tracking for, monitors, and analyzes A/B tests and iterative experiments to drive data-driven product improvements. It starts with experiment discovery and planning: analyzing existing product metrics to find optimization opportunities, defining clear hypotheses with measurable success criteria, calculating appropriate sample sizes via statistical power calculations, and building an experiment timeline with milestones.

Experiment design covers control and variant configurations, primary and secondary metrics, statistical significance thresholds (typically 95% confidence), and a randomization strategy for unbiased user assignment. Implementation tracking generates tracking schemas for experiment events, monitoring dashboards for real-time experiment health, automated alerts for anomalies or technical issues, and implementation requirements for the development team.

Monitoring and analysis tracks experiment progress and statistical significance daily, detects issues like sample-ratio mismatches or external confounding factors, performs interim analyses against early-stopping criteria, and generates automated performance reports. Results and recommendations calculate statistical and practical significance, analyze segmented results across user cohorts, document insights with a clear go/no-go recommendation, and plan follow-up experiments based on what was learned.

Deliverables include an Experiment Plan (name, hypothesis, primary/secondary metrics, calculated sample size, duration, success criteria), a tracking implementation schema for experiment events, and a Results Report (status, statistical significance with p-value, effect size with confidence interval, launch/don't-launch/iterate recommendation, key insights, and next steps). Guidelines followed throughout: proper sample-size calculation and no early peeking at results, weighing practical significance alongside statistical significance, segmenting results to catch nuanced effects, accounting for seasonality and external factors, thorough documentation for future reference, automated monitoring to reduce manual oversight, and ethical testing that doesn't harm user experience or violate privacy.

Use this agent when running A/B tests or iterative experiments that need proper statistical design - sample sizing, significance thresholds, and unbiased randomization - plus real-time monitoring and a clear go/no-go recommendation at the end. It is well suited to product teams with an existing analytics/tracking setup to instrument. It is not meant for decisions that don't need a controlled experiment (e.g. a clear bug fix or obvious UX improvement) - reserve it for genuine A/B or multivariate tests where the outcome is uncertain.

Input: existing product metrics, a hypothesis, and the metric(s) to optimize.

Output: an Experiment Plan, a tracking implementation schema, and a Results Report with a statistical recommendation. Example tracking schema the agent produces:

// Event tracking schema
{
  "experiment_id": "exp_123",
  "user_id": "user_456",
  "variant": "control|treatment",
  "event_type": "assignment|conversion|interaction",
  "timestamp": "2024-01-01T12:00:00Z",
  "metadata": {}
}

Produces tracking schemas and monitoring dashboard specs meant to be implemented in an existing analytics/event-tracking stack; it does not connect to a specific experimentation platform itself.

Product managers and growth/data teams running A/B tests who need proper statistical rigor (sample sizing, significance testing, segmentation) and a clear launch recommendation rather than an ad hoc read of raw metrics.