Design Rigorous A/B Tests

This skill enforces a gated, rigorous process for setting up A/B tests, designed to prevent peeking, ensure statistical power, and block invalid hypotheses before any code is written. A valid hypothesis needs an observation or evidence base, a single specific change, a directional expectation, a defined audience, and measurable success criteria. Before designing variants or metrics, the hypothesis must be locked - presented with target audience, primary metric, expected direction, and Minimum Detectable Effect, with explicit confirmation before proceeding. A mandatory assumptions check covers traffic stability, user independence, metric reliability, randomization quality, and external factors like seasonality or campaigns; weak or violated assumptions mean warning the user and recommending delay or redesign. Test type defaults to a simple A/B test unless there's a clear reason for A/B/n (multiple variants, more traffic needed), multivariate (interaction effects, very high traffic), or split URL testing (major structural changes). Metrics split into a single frozen primary metric tied directly to the hypothesis, secondary metrics for context that must never override the primary, and guardrail metrics that must not degrade and trigger a test stop if they move significantly negative. Sample size and duration require a baseline rate, MDE, significance level (typically 95%), and statistical power (typically 80%) before proceeding.

Before the execution readiness gate, a five-item tracking verification checklist makes "tracking is verified" concrete: confirm each metric-dependent event actually fires and lands in analytics within 30 seconds, verify the variant assignment ID is attached to every fired event (not just the entry event, checked across 5+ sample events per variant), confirm reloads don't double-count events (with the variant ID part of any client-side dedup key), confirm the first 100 assignment records split within plus-or-minus 5% of the configured allocation, and confirm every guardrail metric has a working dashboard or alert. The execution readiness gate is a hard stop: implementation may only begin once the hypothesis is locked, the primary metric is frozen, sample size is calculated, duration is defined, guardrails are set, and tracking is verified - any missing item stops the process.

Use this before setting up any A/B test, to force hypothesis rigor, statistical validity, and tracking correctness ahead of implementation.

During the test: monitor technical health and document external factors, but never stop early because results "look good," change variants mid-test, add new traffic sources, or redefine success criteria. When analyzing results: never generalize beyond the tested population, never claim causality beyond the tested change, never override a guardrail failure even if the primary metric wins, and keep statistical significance separate from business judgment - a significant positive result supports rollout, a significant negative rejects the variant, an inconclusive result calls for more traffic or a bolder change, and any guardrail failure means do not ship regardless of the primary result. The skill explicitly refuses to proceed if the baseline rate can't be estimated, traffic is insufficient to detect the MDE, the primary metric is undefined, multiple variables are being changed without a proper design, or the hypothesis can't be clearly stated - explaining why and recommending next steps rather than proceeding anyway.

Input is a user problem, access to an analytics source, and a rough traffic-volume estimate. Output is a locked hypothesis, a defined primary/secondary/guardrail metric set, a calculated sample size and duration, a verified tracking setup, and ultimately a documented test record (hypothesis, variants, metrics, sample size versus achieved, results, decision, learnings, follow-up ideas) stored in a shared, searchable location to avoid repeating past mistakes.

Assumes an existing analytics/event-tracking stack and an assignment/randomization system whose output (event firing, variant attribution, dedup behavior, split accuracy) the tracking-verification checklist directly inspects.

Product managers, growth teams, and data scientists setting up A/B tests who need enforced rigor - locked hypotheses, verified tracking, and a hard readiness gate - rather than an ad hoc test launch.