Design Rigorous A/B Tests
A rigorous A/B test setup skill with hard gates for hypothesis, sample size, tracking verification, and execution readiness.
Why it matters
Ensure every A/B test is valid, rigorous, and safe before implementation, preventing common pitfalls like 'peeking' and enforcing statistical power.
Outcomes
What it gets done
Validate hypotheses against a quality checklist.
Define primary, secondary, and guardrail metrics.
Calculate sample size and estimate test duration.
Perform assumptions and validity checks before proceeding.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/ag-ab-test-setup | bash Overview
A/B Test Setup
A gated A/B test setup skill enforcing hypothesis lock, assumption checks, sample size calculation, a tracking verification checklist, and a hard execution readiness gate before implementation. Use before setting up any A/B test to enforce statistical rigor; never stop a test early, change variants mid-test, or override a guardrail failure even if the primary metric wins.
What it does
This skill enforces a gated, rigorous process for setting up A/B tests, designed to prevent peeking, ensure statistical power, and block invalid hypotheses before any code is written. A valid hypothesis needs an observation or evidence base, a single specific change, a directional expectation, a defined audience, and measurable success criteria. Before designing variants or metrics, the hypothesis must be locked - presented with target audience, primary metric, expected direction, and Minimum Detectable Effect, with explicit confirmation before proceeding. A mandatory assumptions check covers traffic stability, user independence, metric reliability, randomization quality, and external factors like seasonality or campaigns; weak or violated assumptions mean warning the user and recommending delay or redesign. Test type defaults to a simple A/B test unless there's a clear reason for A/B/n (multiple variants, more traffic needed), multivariate (interaction effects, very high traffic), or split URL testing (major structural changes). Metrics split into a single frozen primary metric tied directly to the hypothesis, secondary metrics for context that must never override the primary, and guardrail metrics that must not degrade and trigger a test stop if they move significantly negative. Sample size and duration require a baseline rate, MDE, significance level (typically 95%), and statistical power (typically 80%) before proceeding.
Before the execution readiness gate, a five-item tracking verification checklist makes "tracking is verified" concrete: confirm each metric-dependent event actually fires and lands in analytics within 30 seconds, verify the variant assignment ID is attached to every fired event (not just the entry event, checked across 5+ sample events per variant), confirm reloads don't double-count events (with the variant ID part of any client-side dedup key), confirm the first 100 assignment records split within plus-or-minus 5% of the configured allocation, and confirm every guardrail metric has a working dashboard or alert. The execution readiness gate is a hard stop: implementation may only begin once the hypothesis is locked, the primary metric is frozen, sample size is calculated, duration is defined, guardrails are set, and tracking is verified - any missing item stops the process.
When to use - and when NOT to
Use this before setting up any A/B test, to force hypothesis rigor, statistical validity, and tracking correctness ahead of implementation.
During the test: monitor technical health and document external factors, but never stop early because results "look good," change variants mid-test, add new traffic sources, or redefine success criteria. When analyzing results: never generalize beyond the tested population, never claim causality beyond the tested change, never override a guardrail failure even if the primary metric wins, and keep statistical significance separate from business judgment - a significant positive result supports rollout, a significant negative rejects the variant, an inconclusive result calls for more traffic or a bolder change, and any guardrail failure means do not ship regardless of the primary result. The skill explicitly refuses to proceed if the baseline rate can't be estimated, traffic is insufficient to detect the MDE, the primary metric is undefined, multiple variables are being changed without a proper design, or the hypothesis can't be clearly stated - explaining why and recommending next steps rather than proceeding anyway.
Inputs and outputs
Input is a user problem, access to an analytics source, and a rough traffic-volume estimate. Output is a locked hypothesis, a defined primary/secondary/guardrail metric set, a calculated sample size and duration, a verified tracking setup, and ultimately a documented test record (hypothesis, variants, metrics, sample size versus achieved, results, decision, learnings, follow-up ideas) stored in a shared, searchable location to avoid repeating past mistakes.
Integrations
Assumes an existing analytics/event-tracking stack and an assignment/randomization system whose output (event firing, variant attribution, dedup behavior, split accuracy) the tracking-verification checklist directly inspects.
Who it's for
Product managers, growth teams, and data scientists setting up A/B tests who need enforced rigor - locked hypotheses, verified tracking, and a hard readiness gate - rather than an ad hoc test launch.
Source README
A/B Test Setup
1️⃣ Purpose & Scope
Ensure every A/B test is valid, rigorous, and safe before a single line of code is written.
- Prevents "peeking"
- Enforces statistical power
- Blocks invalid hypotheses
2️⃣ Pre-Requisites
You must have:
- A clear user problem
- Access to an analytics source
- Roughly estimated traffic volume
Hypothesis Quality Checklist
A valid hypothesis includes:
- Observation or evidence
- Single, specific change
- Directional expectation
- Defined audience
- Measurable success criteria
3️⃣ Hypothesis Lock (Hard Gate)
Before designing variants or metrics, you MUST:
- Present the final hypothesis
- Specify:
- Target audience
- Primary metric
- Expected direction of effect
- Minimum Detectable Effect (MDE)
Ask explicitly:
“Is this the final hypothesis we are committing to for this test?”
Do NOT proceed until confirmed.
4️⃣ Assumptions & Validity Check (Mandatory)
Explicitly list assumptions about:
- Traffic stability
- User independence
- Metric reliability
- Randomization quality
- External factors (seasonality, campaigns, releases)
If assumptions are weak or violated:
- Warn the user
- Recommend delaying or redesigning the test
5️⃣ Test Type Selection
Choose the simplest valid test:
- A/B Test - single change, two variants
- A/B/n Test - multiple variants, higher traffic required
- Multivariate Test (MVT) - interaction effects, very high traffic
- Split URL Test - major structural changes
Default to A/B unless there is a clear reason otherwise.
6️⃣ Metrics Definition
Primary Metric (Mandatory)
- Single metric used to evaluate success
- Directly tied to the hypothesis
- Pre-defined and frozen before launch
Secondary Metrics
- Provide context
- Explain why results occurred
- Must not override the primary metric
Guardrail Metrics
- Metrics that must not degrade
- Used to prevent harmful wins
- Trigger test stop if significantly negative
7️⃣ Sample Size & Duration
Define upfront:
- Baseline rate
- MDE
- Significance level (typically 95%)
- Statistical power (typically 80%)
Estimate:
- Required sample size per variant
- Expected test duration
Do NOT proceed without a realistic sample size estimate.
Tracking Verification (Required before Gate 8)
Before entering the Execution Readiness Gate below, run through this checklist to make "Tracking is verified" mean something concrete:
- Event firing: Trigger each event the primary and secondary metrics depend on (sign-up, add-to-cart, custom event) on staging or a debug page, and confirm it lands in your analytics destination within 30 seconds.
- Variant attribution: Verify that the variant assignment ID is attached to every fired event - not just the entry event. Use your analytics' raw event view to compare a sample of 5+ events per variant.
- De-duplication: Confirm that a user reloading the page does not cause double-counted events. If your stack uses client-side de-duping, the variant ID must be part of the dedup key.
- Sample randomization: Pull the first 100 assignment records from your assignment table; the variant split should be within ±5% of the configured allocation.
- Guardrail metric pipeline: Each guardrail metric defined in §6️⃣ must have a working dashboard or alert by the time the test launches.
If any of the above fails, stop and resolve it before Gate 8.
8️⃣ Execution Readiness Gate (Hard Stop)
You may proceed to implementation only if all are true:
- Hypothesis is locked
- Primary metric is frozen
- Sample size is calculated
- Test duration is defined
- Guardrails are set
- Tracking is verified
If any item is missing, stop and resolve it.
Running the Test
During the Test
DO:
- Monitor technical health
- Document external factors
DO NOT:
- Stop early due to “good-looking” results
- Change variants mid-test
- Add new traffic sources
- Redefine success criteria
Analyzing Results
Analysis Discipline
When interpreting results:
- Do NOT generalize beyond the tested population
- Do NOT claim causality beyond the tested change
- Do NOT override guardrail failures
- Separate statistical significance from business judgment
Interpretation Outcomes
| Result | Action |
|---|---|
| Significant positive | Consider rollout |
| Significant negative | Reject variant, document learning |
| Inconclusive | Consider more traffic or bolder change |
| Guardrail failure | Do not ship, even if primary wins |
Documentation & Learning
Test Record (Mandatory)
Document:
- Hypothesis
- Variants
- Metrics
- Sample size vs achieved
- Results
- Decision
- Learnings
- Follow-up ideas
Store records in a shared, searchable location to avoid repeated failures.
Refusal Conditions (Safety)
Refuse to proceed if:
- Baseline rate is unknown and cannot be estimated
- Traffic is insufficient to detect the MDE
- Primary metric is undefined
- Multiple variables are changed without proper design
- Hypothesis cannot be clearly stated
Explain why and recommend next steps.
Key Principles (Non-Negotiable)
- One hypothesis per test
- One primary metric
- Commit before launch
- No peeking
- Learning over winning
- Statistical rigor first
Final Reminder
A/B testing is not about proving ideas right.
It is about learning the truth with confidence.
If you feel tempted to rush, simplify, or “just try it” -
that is the signal to slow down and re-check the design.
When to Use
This skill is applicable to execute the workflow or actions described in the overview.
Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.