Design and run statistically valid A/B tests

Walks through the full lifecycle of an experiment: writing a hypothesis in the structured form "Because [observation], we believe [change] will cause [outcome] for [audience], we'll know this is true when [metrics]"; choosing a test type from a table of A/B, A/B/n, MVT, and Split URL tests by traffic needed; and selecting a primary metric, secondary metrics, and guardrail metrics (illustrated with a pricing-page example: plan selection rate as primary, time-on-page and plan distribution as secondary, support tickets and refund rate as guardrails). It covers designing variants across headlines/copy, visual design, CTA, and content, and choosing traffic allocation (50/50 standard, 90/10 or 80/20 conservative, or a ramping rollout). Implementation is split into client-side (JavaScript modifies the page after load, quick but can flicker) and server-side (variant determined before render, no flicker, needs dev work). A pre-launch checklist, a during-test do/avoid list, and an explicit warning against "the peeking problem" (stopping early on partial data) guard test integrity. Analysis covers statistical significance (95% confidence, p < 0.05), effect size versus the minimum detectable effect, guardrail checks, and segment differences (mobile vs. desktop, new vs. returning). Every test is documented with a hypothesis, variants, results, and decision.

Beyond single tests, it also defines a continuous growth-experimentation program: an experiment loop (generate hypotheses, prioritize, run, analyze, promote winners to a playbook, generate new hypotheses); an ICE score (Impact + Confidence + Ease) / 3 for ranking the backlog; velocity targets (4-8 experiments/month, 20-30% win rate, 2-4 week average duration, 20+ hypotheses queued); and a playbook template that captures each winning pattern and where else it could apply, reviewed on a weekly/bi-weekly/monthly/quarterly cadence.

Use it when the task is to plan, design, or implement an A/B test, split test, multivariate test, or a broader growth-experimentation program - triggered by phrases like "A/B test," "split test," "experiment," "variant copy," "hypothesis," or "should I test this." Before asking clarifying questions it checks for an existing product-marketing context file (.agents/product-marketing.md, or the legacy product-marketing-context.md) and reads that first.

For adjacent jobs in the same cycle, the source points to separate skills instead: cro for generating test ideas from CRO principles, analytics for setting up test measurement, and copywriting for writing the variant copy itself. The skill's own metadata marks it risk: critical, and its stated limitations apply here directly: verify commands, generated code, dependencies, credentials, and external service behavior before applying changes, and do not treat its templates or examples as a substitute for environment-specific testing, security review, or user approval before destructive or costly changes.

It starts by gathering test context (what's being improved), current state (baseline conversion rate, current traffic volume), and constraints (technical complexity, timeline, available tools). From there it uses sample-size reference tables (indexed by baseline conversion rate and the lift you want to detect) and a duration formula (sample per variant x number of variants / (daily traffic x % exposed)) to size and schedule the test, checked against a pre-launch checklist. The outputs are the filled-in artifacts: a test plan (hypothesis, variant details, metrics, segment-analysis plan, success criteria), a results-documentation report (sample size versus target, primary/secondary/guardrail metric tables with confidence intervals, segment breakdowns, decision and learnings), and, for a winning test, a new entry in the reusable experiment playbook.

Names specific sample-size calculators (Evan Miller's, Optimizely's, AB Test Guide, and VWO's duration calculator) and experimentation platforms by role: PostHog, Optimizely, and VWO for client-side implementation; PostHog, LaunchDarkly, and Split for server-side implementation; and PostHog's Bayesian approach, Optimizely's Stats Accelerator, and VWO's SmartStats specifically for sequential testing that lets you check results before the full sample size is reached.

Growth, product, and marketing teams running conversion-rate-optimization or growth-experimentation work, from a single test to an ongoing ICE-prioritized experimentation cadence. Sourced from the community marketingskills repository (coreyhaines31/marketingskills) under the MIT license.