Designing and reading A/B tests

Skills you'll practise

• Write a testable hypothesis with one primary metric chosen before launch
• Choose a randomization unit and explain how the wrong one contaminates a test
• Calculate absolute and relative lift from clearly defined test counts
• Check a test for sample ratio mismatch before reading its results
• Explain why stopping early on a good-looking result and slicing many segments produce false winners
• Write the limitations of a test result and recommend a next action

Course outline

1. 1.Designing and reading A/B testsVideo · 2 min
2. 2.Planning a test that can answer a questionLesson · 17 min
3. 3.Running the test fairlyLesson · 16 min
4. 4.Reading results and stating limitationsLesson · 17 min
5. 5.Designing and reading A/B tests: knowledge checkKnowledge check · 18 questions
6. 6.Day three of a four-week testScenario
7. 7.Final exam9 questions · passing it completes the course, so people who already know the material can test out

Sources it draws on

The lessons and questions are written from these references, so learners can go back to the original.

• Kohavi, Tang and Xu: Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (Cambridge University Press, 2020)
• Evan Miller: How Not To Run an A/B Test (repeated significance testing)
• NIST/SEMATECH e-Handbook of Statistical Methods, 1.3.5.15: Chi-square goodness-of-fit test
• General marketing and sales practice (starter content; have a subject expert review it)