Amplitude AI Analytics

Build an AI Growth Optimization System

Amplitude

Eppo

Cameo

SpotHero

PowerReviews

Eric Metelka

Product Leader, Amplitude Experiment |
Lenny's Top 25 Contributor

Recognize where evals stop giving signal and when to run a live experiment

Identify where evals end and experiments begin

Apply a practical framework for sequencing eval checks, online A/B tests, and Amplitude analytics across the AI product

Layer evals, experiments, and analytics

Use AI-native tooling to generate hypotheses, run tests, and surface analysis without manual overhead at each step.

Automate your experimentation loop

Learn proven approaches for quickly improving AI applications.  Build AI that works better than the competition, regardless of the use-case.

ML Engineer with 20 years of experience

Airbnb

GitHub

DataRobot

AlixPartners

Hamel Husain

ML Engineer with 20 years of experience.

ML Systems & Applied AI Evals Researcher

Google

UC Berkeley

Stanford University

Shreya Shankar

ML Systems Researcher Making AI Evaluation Work in Practice

Cohort 5

Cohort 6

🚨 September 2026 cohort's material is completely refreshed. 🚨All students get:<ul><li>♾️ Unlimited access to future cohorts &amp; office hours: never worry about timing or missing new material.</li><li>🗄️ Lifetime access to all materials!</li><li>🤖 6 months of unlimited access to our new AI Eval Assistant (more info below).</li><li>🧑‍🏫 10+ hours of office hours to maximize the value of live interaction.</li><li>🏫 A Discord community with continuous access to instructors to get unstuck (even after the course!).</li></ul>---Do you catch yourself asking any of the following questions while building AI applications?1. How do I test applications when the outputs require subjective judgements?2. If I change the prompt, how do I know I'm not breaking something else?3. Where should I focus my engineering efforts? Do I need to test everything?4. What if I have no data or customers, where do I start?5. What metrics should I track? What tools should I use?6. Can I automate testing and evaluation? If so, how do I trust it?If so, this course is for you.All sessions are live and recorded.

Stop guessing if your AI works. Build the feedback loops that make it better.

How To Collect Data For Evals

Get immediate clarity and direction with Error Analysis

Implement Effective Evaluations

Master Architecture-Specific Eval Strategies

Run Evals In Production

Ensure That Evals Lead To High ROI

Engineers and PMs who ship prompt changes and hope nothing breaks. (You'll learn to measure impact before and after every change.)

Teams still spot-checking AI outputs by hand instead of measuring systematically. (You'll learn how build automated evals you can trust.)

Leaders who don't know where their AI is failing or where to invest resources. You'll learn how to systematically find &amp; prioritize issues.

Learn how to evaluate, ship, and monitor GenAI products that perform reliably in production.

Cohort 4

Head of AI Applied Science

Shopify

Carvana

Harvard University

Stella Liu

Cofounder, AI Evals & Analytics

System1

Uptake

Seasalt.ai

UCLA

University of Washington

Amy Chen

Cofounder at AI Evals & Analytics

Cohort 3

<blockquote>Building AI is easier than ever. Building AI that is reliable, safe, and trusted is not.AI evaluations help teams measure quality, detect failures, and build guardrails around your GenAI systems that will survive in the real world.</blockquote><blockquote>🔹</blockquote><h3>Who this course is for</h3>Data scientists, PMs and QA/QE professionals who:<ul><li>have no prior background in AI evals, or</li><li>have tried LLM-as-a-judge but want a clearer framework.</li></ul><blockquote>🔹How the course works</blockquote><ul><li>Self-paced lectures you can watch anytime</li><li>Hands-on exercises using a pre-built GenAI product</li><li>Dedicated office hours for questions and guidance</li></ul><blockquote>🔹</blockquote><h3>What makes this course different</h3><ul><li>Playbook-driven approach</li><li>Evaluation-driven development</li><li>Product-focused</li><li>Hands-on experience</li><li>Flexible learning + expert support</li></ul>🔹Curious about the ideas behind this course?You can explore related topics in the <a target="_blank" rel="noopener noreferrer nofollow" href="https://datasciencexai.substack.com/">Data Science × AI Substack</a>, where we share perspectives on evaluating real GenAI products.🔹💰 Purchasing Power Parity pricing adjustments upon request. ✉️ Email us: StellaxAmy@gmail.com

Learn How to Evaluate GenAI Products Confidently in 3 Weeks

Execution-ready Testing & Monitoring Playbook

GenAI Risk Management Strategy

Hands-On AI Evals Exercise

Product LeadersBuilding GenAI products and need a clear playbook for evaluation frameworks, success metrics, and shipping with confidence.

Data ScientistsLeveling up with AI product skills: learn AI evals, LLM-as-a-judge, and design metrics for GenAI systems.

QA and Test EngineerAdapting traditional testing practices to GenAI systems.

Build a full AI evals suite in 3 hours on a real use case

Founder @Lovelaice | 10 years in Product

Finastra

Trading Central

Printify

Madalina Turlea

Product leader with 10+ years in platform and SaaS products | Founder Lovelaice

Founder @ Lovelaice | 2x founder & CTO

Teleclinic

Nilo

SIXT Group

Freeletics

Catalina Turlea

2x founder & CTO | 14+ years in tech | Founder @Lovelaice

June

<blockquote>Whether you're building your first AI feature or exploring what AI could do in your product, there's a moment every PM hits: you look at the AI output and think "this seems okay?" but you don't have a structured way to know for sure.You've heard about evals. You know vibe checking isn't enough. But where do you actually start?This workshop takes you from that starting point to a working evaluation suit, built on a real AI feature use case, in 3 hours.You'll write a production system prompt, run it across multiple LLMs, read and annotate real AI responses to understand where they break, find failure patterns through error analysis, define automatic evals based on what you found, and measure whether your improvements actually worked.You'll leave with hands on experience on building your own evals suite, failure taxonomy, and a repeatable process to evaluate and improve any AI feature from here.Every participant works hands-on annotating AI responses, writing evals and running improvement iterations. We've run 1,500+ AI experiments with product teams and this workshop is the compressed version of what we've seen work.</blockquote>

Design, build, validate and iterate on an AI feature, hands-on

Understand the most important components of building AI features at scale

Build an evaluation suite to scale your testing

Learn how to identify failure patterns. Then iterate on both prompt and evals

The PM who wants hands-on experience on how to build high performing AI features

The PM who is overwhelmed by the technical tools and process around building AI

The product lead who wants improve the iteration speed or wants to scale manual testing with automatic evals

Evals aren't your product moat. A continuous improvement flywheel is.

AI Founder & Advisor to F500s | Ex-AWS

Amazon Web Services

Microsoft

University of Oxford

Aishwarya Naresh Reganti

June Cohort

Check out our free AI Evals Certification <a target="_blank" rel="noopener noreferrer nofollow" href="https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/free_courses/ai_evals_for_everyone/README.md">here</a> (5000+) Learners!Most teams building AI products today know about evals and LLM judges, but they’re deeply confused about how to actually turn them into actionable improvement signals post production. Just throwing in LLM judges won’t magically fix your product. 🚨 You can also upgrade to our 1-year <a target="_blank" rel="noopener noreferrer nofollow" href="https://levelup-labs.ai/ai-builder-pass">AI Builder Pass</a>, which includes unlimited retakes, access to all our other courses, live sessions and partner credits from companies like Vapi AI, Arize, Cursor, Vercel, OpenAI, and more! Register for this course, and we’ll send you upgrade details over email.<ul><li>This course shows how to use offline evals, online evals, and production monitoring to drive continuous improvement, based on real production experience.</li><li>This course is for people who’ve already built AI systems and are stuck with evals that are noisy, expensive, or uselessIf you need something more foundational, check out our top-rated flagship<a target="_blank" rel="noopener noreferrer nofollow" href="https://maven.com/aishwarya-kiriti/genai-system-design"> enterprise AI course</a> taken by over 1500+ builders and leaders.</li><li>We follow a flipped-classroom format with async lecture content and meet twice a week for office hour style sessions to optimize two-way/interactive live time.</li></ul>

Move beyond one-time evals to continuous improvement flywheels for AI products

Understand what evals actually mean and why it’s misunderstood

Build a pre-deployment evaluation foundation from scratch

Choose the right types of metrics for the system you’re building

Monitor in production and close the improvement loop

You should have built small AI applications before and be familiar with concepts like RAG, MCP, and tool use.

A good understanding of AI system design

The course includes optional Python assignments, you're free to use coding agents, but basic coding skills are recommended

Working knowledge of Python/Coding (Optional)

Software/AI Engineers, Strategists, Data Professionals, Solution Architects and Consultants looking for systematic evaluation practices

Business Leaders and Product Managers seeking to gain the technical understanding of AI Product Lifecycle and intentional iteration

Founders &amp; teams stuck at systematic evaluations and monitoring, unsure how to turn signals into real product improvements.

The Agent-Powered Super IC

Agentic AI

Coding with AI

AI Workflows

Claude Code

OpenClaw

Vibe Coding

AI Evals

AI Transformation

RAG & Search

AI for PMs

AI for Engineers

AI for Designers

AI for Marketers

AI for Founders

Product Sense

Product Discovery

User Research

Prototyping

Growth

Analytics

Tech Foundations

Strategy

Influence

Leadership

Career Growth

Machine Learning

LLM Ops

Context Eng

Security

System Design

Figma

Design Systems

Visual Design

Design Strategy

Automation

Content Marketing

Demand Gen

Go-to-Market

Product Marketing

Positioning

Social Media

Brand

B2B Marketing

SEO & AEO

AI for Leaders

AI Governance

Communication

Management

People Operations

Exec Presence

Storytelling

Goal-setting

Personal Brand

Fundraising

Everyone

Operators

Data Scientists

Business Analysts

User Researchers

Customer Success

Project Managers

HR Professionals

Sales People

Lawyers

Finance

Investors

Real Estate

Educators

Creators

Product managers who drive strategy, development, and launch of products.

Data scientists who apply statistical and ML techniques to extract insights from data.

Project managers who organize and oversee execution of initiatives, timelines, and cross-functional delivery.

Applying AI tools and techniques to professional workflows — assign alongside more specific AI topics.

Using data and metrics to measure performance, run experiments, and inform decisions.

Driving user acquisition, activation, retention, and monetization through product-led growth strategies.

Measuring and evaluating AI model performance, quality, and reliability for production use cases.

Automating professional workflows with AI tools — productivity, operations, and task automation.

Build an AI Growth Optimization System

What you'll learn

Why this topic matters

You'll learn from

Eric Metelka