Interpret A/B test results without fooling yourself
You are an experienced experimentation analyst. Help me interpret this A/B test honestly. Hypothesis: [e.g. "A shorter checkout form will increase completed purchases"] Primary metric: [e.g. "conversion rate = purchases / visitors who reached checkout"] Secondary and guardrail metrics: [e.g. "average order value", "refund rate", "page load time"] Planned design: [e.g. "50/50 split, planned sample of 20,000 per variant, 2 weeks, two-sided test, alpha 0.05"] What actually happened: [e.g. "ran 16 days; we checked results daily"] Results: | Variant | Users | Conversions | [OTHER METRICS] | | Control | [N] | [N] | | | Treatment | [N] | [N] | | Please: 1. Calculate the conversion rate per variant, the absolute and relative difference, and a 95% confidence interval for the difference. Show the formula and the numbers so I can verify them. 2. Check for pitfalls: sample ratio mismatch (is the split close to plan?), peeking or stopping early, multiple metrics or segments tested, novelty effects, too short a run to cover weekly cycles. 3. Look at guardrail metrics for harm. 4. Give a recommendation: ship, do not ship, or run longer, with the main reason. 5. Write a 3-sentence summary for non-technical stakeholders, including uncertainty. Do not claim certainty the data cannot support. If a calculation needs information I did not give, say so.
Works with: ChatGPT, Claude, Gemini