A/B testing without fooling yourself
For marketing analysts, product managers and anyone who reads A/B test results and has to say whether a change really worked. Complete 'Designing and reading A/B tests' first: it covers hypotheses, primary metrics, randomisation, lift, sample ratio checks and peeking. This course goes underneath the dashboard: you'll calculate the uncertainty in a conversion rate and a confidence interval for a difference, run a two-proportion z test by hand, see how sample size depends on the effect you want to detect and plan duration, and recognise how many variants, metrics and segments manufacture false winners. You need a calculator or spreadsheet with a square root function. It uses the NIST/SEMATECH e-Handbook of Statistical Methods and GOV.UK guidance on A/B testing.
- Level
- Advanced
- Length
- About 72 minutes
- Contents
- 4 lessons · final exam
- Status
- Published · updated 10 Oct 2026
Skills you'll practise
- Calculate the standard error of a conversion rate and a confidence interval for the difference between two variants
- Calculate and interpret a two-proportion z statistic for an A/B test
- Estimate how the required sample size changes with the effect you want to detect, and plan the test's duration
- Identify how multiple variants, metrics and segments create false winners, and choose a fair decision
Course outline
- 1.Calculating a standard error and a confidence interval for the differenceLesson · 16 min
- 2.Calculating and interpreting a two-proportion z statisticLesson · 16 min
- 3.Estimating how sample size changes with the effect to detect, and planning durationLesson · 14 min
- 4.Identifying how multiple variants, metrics and segments create false winnersLesson · 14 min
- 5.A/B testing without fooling yourself: knowledge checkKnowledge check · 14 questions
- 6.A/B testing without fooling yourself: practical exerciseKnowledge check · 1 question
- 7.Final exam6 questions · passing it completes the course, so people who already know the material can test out
Sources it draws on
The lessons and questions are written from these references, so learners can go back to the original.
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.1.3: What are statistical tests?
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.1.4: What are confidence intervals?
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.2.4: Does the proportion of defectives meet requirements?
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.2.4.2: Sample sizes required
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.3.3: How can we determine whether two processes produce the same proportion of defectives?
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.4.7: How can we make multiple comparisons?
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.4.7.3: Bonferroni's method
- GOV.UK (Office for Health Improvement and Disparities): A/B testing: comparative studies
See it with your own jobs and topics
Tell us about your team and we'll walk you through setup, from choosing jobs to your first skills check.