Staging environment
Shane Butler
Sravya Madipalli
Hai Guan
Free Lesson

Agentic Experiment Design & Analysis

Part of The Agent-Powered Super IC

60 min
Jun 5, 2026 3:00 PM

What you'll learn

Frame experiments using the Agentic Loop

Apply the 4 stages (Frame, Power, Validate, Decide) to design tests that hold up to scrutiny.

Catch the validity gaps that ship bad decisions

Spot sample ratio mismatch, novelty effects, and underpowered tests before the readout meeting.

Use AI agents to design and analyze A/B tests

Apply a design helper prompt and watch Claude Code produce a full experiment readout from raw data.

Why this topic matters

Most experiment readouts ship bad decisions: p-values without validity checks, no segment analysis, vague recommendations. AI makes it easier to crunch numbers, which makes it easier to crunch the wrong ones. Learn the Agentic Experiment Loop to design, validate, and analyze tests, then see how AI agents accelerate every stage without replacing judgment.

You'll learn from

Shane Butler

Shane Butler

Principal Data Scientist at Ontra

Sravya Madipalli

Sravya Madipalli

Senior DS Leader (Ex-Microsoft)

Hai Guan

Hai Guan

Head of Data at Ontra, Ex-LinkedIn

See all products from AI Analyst Lab
Get free access