Design and run statistically valid A/B tests

A/B Test Setup turns a proposed conversion change into a test with a documented hypothesis, measurable outcome, and decision process. It starts with baseline conversion, page traffic, the intended change, existing tools, and timeline constraints. Existing product-marketing context is read before planning so the experiment reflects the product and audience.

The hypothesis connects an observed problem to a specific change, expected outcome, audience, and measurement. The workflow favors one meaningful change at a time and distinguishes a primary business metric from secondary metrics that explain behavior and guardrails that should not deteriorate. A pricing-page example uses plan selection as the primary metric, time on page and plan distribution as secondary metrics, and support tickets and refunds as guardrails.

Design covers A/B, A/B/n, multivariate, and split-URL tests, with traffic requirements increasing for more complex comparisons. Sample planning uses baseline conversion and the minimum detectable effect, supported by references/sample-size-guide.md and calculators such as Evan Miller and Optimizely. Variant ideas include copy, visual hierarchy, CTA wording or placement, and content ordering, selected to match the hypothesis.

Use it when planning, designing, or implementing an experiment, or when establishing a continuous growth testing program. Avoid starting with an undetectably small change, combining unrelated changes, or running without a hypothesis and verified measurement.

Predetermine the sample size and avoid repeatedly inspecting results to stop at a favorable moment. Keep variants and traffic sources stable during the run. Monitoring technical problems and significant guardrail harm remains necessary; a harmful test can be stopped without declaring an early winner. Treat results below the planned sample as preliminary, and avoid cherry-picking segments or overstating inconclusive findings.

Inputs include current conversion, traffic volume, the proposed change and rationale, the smallest useful improvement, available testing tools, and previous experiments in the area. The plan specifies the hypothesis, primary metric, sample size, variants, allocation, and tracking checks. Pre-launch QA verifies each variant and its instrumentation.

A standard allocation is 50/50. Conservative 90/10 or 80/20 splits limit exposure, while ramping starts small for technical risk. Returning users should retain their variant, with exposure balanced across time periods.

Analysis checks sample completion, confidence intervals, practical effect size relative to MDE, secondary metrics, guardrails, and segment differences. Outcomes lead to implementing a winner, retaining the control, investigating mixed signals, or planning a stronger follow-up. Documentation captures screenshots, sample counts, metrics, significance, decisions, and learning using references/test-templates.md.

For ongoing experimentation, generate hypotheses from analytics, research, competitors, support, recordings, and earlier tests. Score Impact, Confidence, and Ease from 1 to 10, average them for ICE, and re-score monthly. Track velocity, win rate, duration, backlog depth, and cumulative lift. A playbook records results, guardrails, segment deltas, explanations, reusable patterns, and implementation status. Weekly reviews monitor running tests; biweekly reviews close completed tests; monthly reviews replenish and prioritize the backlog; quarterly audits look for patterns to scale and undertested funnels.

Client-side approaches modify pages after load and may flicker; tool options include PostHog, Optimizely, and VWO. Server-side assignment happens before rendering, requires development work, and names PostHog, LaunchDarkly, and Split. Related cro, analytics, and copywriting skills support ideas, measurement, and variant copy. These references guide tool selection and workflow; environment-specific implementation and validation remain part of the work.