Free Lesson
A/B Testing Replication Crisis: Lessons from 10 A/B tests
45 min
Mar 4, 2026 1:00 PM
Virtual (Zoom)
In this video
What you'll learn
How to design a trustworthy A/B tests
How we designed the 10 A/B tests to evaluate four patterns: choosing the metrics and duration for proper power.
Most claims of large improvements are highly exaggerated
While we expected to see exaggerations, our results showed surprisingly small effects.
Lessons
Key lessons from the Trustworthy A/B Patterns community project.
Why this topic matters
Claims of large lifts in A/B tests are widespread, yet many are only supported by small online experiments. The Trustworthy A/B Patterns community project (https://bit.ly/trustworthyABPatterns)
was started in June 2024 by Kohavi, Linowski, and Vermeer to provide realistic assessment of several patterns with large-scale A/B tests.
You'll learn from
Ronny Kohavi
Technical fellow and VP at Microsoft & Airbnb, co-author of A/B testing book
Previously at Microsoft, Airbnb, Amazon
