Free Lesson
Design Experiments for AI Features
60 min
Feb 19, 2026 3:00 PM
Virtual (Zoom)
In this video
What you'll learn
Where experiments fit in the AI impact chain
A/B tests are best for linking quality changes to user behavior
Define exposure and randomization correctly
Learn eligibility, exposure, and unit of randomization so “treatment” means users actually experienced the AI.
Know when results are decision-grade
Interpret lift, variance, and segments well enough to decide ship, ramp, hold, or rollback with confidence.
Join AI Builders Slack Community. Link: bit.ly/ai-connect
Learn Agentic Analytics workflows and AI Eval design. Get feedback on what you’re building from other AI builders.
Why this topic matters
A/B tests are often treated as the default proof of AI impact. For AI features, they only work under specific conditions: correct exposure, correct randomization, and the right question. This lesson shows where experiments are strong in the impact chain and what you need for results you can trust.
You'll learn from
Shane Butler
Principal Data Scientist, AI Evaluations at Ontra
Previously at Stripe, Nextdoor, PwC
