AI-first founder since 2016

UC Berkeley

Facebook

Mark Cuban Companies

Deloitte

Haroon Choudery

Founder at Seeko. Writes AI Ready to 50,000+ subs. Previously founded Autoblocks & AI For Anyone

Master Evaluation Techniques for LLM Apps

OpenAI

 Intercom

Retool

Airtable

CEO, Autoblocks AI

Learn what role evaluations play in LLM apps and why they are crucial ensuring their effectiveness & reliability.

Why evaluations are necessary

Explore how to select the best evaluation techniques (i.e., rules-based evals or LLM judges) for your LLM use case

How to choose the right evaluations

Discover methods, like fine-tuning & expert alignment, for improving the consistency and accuracy of your evaluations.

Improving evaluation reliability

Learn proven approaches for quickly improving AI applications.  Build AI that works better than the competition, regardless of the use-case.

ML Engineer with 20 years of experience

Airbnb

GitHub

DataRobot

AlixPartners

Hamel Husain

ML Engineer with 20 years of experience.

ML Systems & Applied AI Evals Researcher

Google

Stanford University

Shreya Shankar

ML Systems Researcher Making AI Evaluation Work in Practice

Cohort 5

Cohort 6

🚨 September 2026 cohort's material is completely refreshed. 🚨All students get:<ul><li>♾️ Unlimited access to future cohorts &amp; office hours: never worry about timing or missing new material.</li><li>🗄️ Lifetime access to all materials!</li><li>🤖 6 months of unlimited access to our new AI Eval Assistant (more info below).</li><li>🧑‍🏫 10+ hours of office hours to maximize the value of live interaction.</li><li>🏫 A Discord community with continuous access to instructors to get unstuck (even after the course!).</li></ul>---Do you catch yourself asking any of the following questions while building AI applications?1. How do I test applications when the outputs require subjective judgements?2. If I change the prompt, how do I know I'm not breaking something else?3. Where should I focus my engineering efforts? Do I need to test everything?4. What if I have no data or customers, where do I start?5. What metrics should I track? What tools should I use?6. Can I automate testing and evaluation? If so, how do I trust it?If so, this course is for you.All sessions are live and recorded.

Stop guessing if your AI works. Build the feedback loops that make it better.

How To Collect Data For Evals

Get immediate clarity and direction with Error Analysis

Implement Effective Evaluations

Master Architecture-Specific Eval Strategies

Run Evals In Production

Ensure That Evals Lead To High ROI

Engineers and PMs who ship prompt changes and hope nothing breaks. (You'll learn to measure impact before and after every change.)

Teams still spot-checking AI outputs by hand instead of measuring systematically. (You'll learn how build automated evals you can trust.)

Leaders who don't know where their AI is failing or where to invest resources. You'll learn how to systematically find &amp; prioritize issues.

Acquire and develop a critical skill for product managers who are leading and contributing to AI products.

GenAI Product Builder @ Google, ex-Meta

Marily Nika, Ph.D AI/ML

Product Manager AI&ML @ Google

Diego Granados

Senior Data, ML and GenAI Scientist

George Zoto

Senior NLP/AI Scientist

Cohort 4

“Does it work?”... “Is it good enough?”... “Can we ship it?”...How do you answer these questions for AI products? You’re responsible for “running evals” but what does that mean?How do you choose the right metrics, interpret fuzzy results, and make a confident decision?This course gives you a framework to do just that.<ul><li>Map user value to evaluation (eval) objectives so your metrics aren’t abstract. Define success then translate it into measurable criteria.</li><li>Choose metrics you can actually maintain: capability, safety, UX friction, latency, cost and “does this reduce support tickets or increase activation.”</li><li>Set ship/no-ship thresholds you can defend to leadership.</li><li>Build lightweight workflows that work in real teams: human review where it matters, automation where it lasts, documentation that drives decisions.</li><li>Consider domain constraints (e.g., healthcare safety) and know what to avoid: silent failures, misleading proxy metrics and tests that don’t reflect production.</li><li>Tie everything to ROI: impact vs unit cost, eval coverage vs reliability, and the minimum viable monitoring you need post-launch.</li></ul>Experience AI evals through a case-based approach with a real AI product that we evaluate together.

Eliminate uncertainty in shipping AI features

Make confident ship or hold decisions for AI features

Translate user value into clear evaluation goals

Choose the right metrics for capability, safety, UX, and cost

Set defensible thresholds leadership will trust

Build lightweight evaluation workflows teams can maintain

Navigate domain constraints and avoid common failures

PMs leading AI features, growth, or platform initiatives

PMs who partner with ML teams and want to set evaluation standards

PMs who need to make clear “ship or hold” calls without doing the engineering

Build robust LLM-powered apps, chatbots, and agents. Learn by writing real code, one line at a time.

CEO @ Actualize

Jay Wengrow

Software Engineer and Educator, Author of A Common-Sense Guide to AI Engineering

Cohort 1

Cohort 2

Do you have a grasp of basic LLM principles but struggle to put it all into code? Tired of scattered documentation, outdated tutorials, and frameworks that impress in demos but collapse in production?You’re not alone. Building with large language models doesn’t have to be confusing - it can be clear, structured, and deeply rewarding.In this course, you’ll build real LLM-powered apps, one line of code at a time. You'll see how an app is built from scratch in real time, and follow along. Along the way, you'll discover not only how to implement these systems, but the why behind each line of code.From chatbots to agents and RAG to evals, you’ll learn the core concepts of building atop LLMs. More importantly, you’ll get how LLMs “think” - allowing you to guide them to do what you want despite their nondeterministic nature. Additionally, you’ll be able to balance quality, latency, and cost, making big-picture decisions about AI-powered app architecture.AI engineering isn’t just another branch of software development - it’s a different mindset altogether.

Gain hands-on AI Engineering proficiency by writing real code

Gain real-world intuition for working with LLMs

Reliably guide LLMs instead of getting unpredictable results.

Build retrieval-augmented assistants that can access proprietary knowledge

Use evals to systematically optimize your LLM app’s performance over time

Assemble agents that act upon the real world

This course is designed for current software engineers who will use Python  to build LLM-powered apps. We won't teach basic coding here.

Software Engineering

We'll be using Python, but we'll keep it simple in case you're more familiar with other coding languages.

Note about Python

If you want an inside look at what's involved with building AI apps but don't plan on coding yourself, you'll still follow what's going on.

Exception: Product Managers

Software engineers who want to build LLM-powered chatbots and agents, or break into AI engineering more generally

Data Scientists/Engineers who want to build their own LLM-powered apps

Product Managers who want to understand AI engineering from the coding perspective

Evals aren't your product moat. A continuous improvement flywheel is.

AI Founder & Advisor to F500s | Ex-AWS

Amazon Web Services

Microsoft

University of Oxford

Aishwarya Naresh Reganti

June Cohort

Check out our free AI Evals Certification <a target="_blank" rel="noopener noreferrer nofollow" href="https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/free_courses/ai_evals_for_everyone/README.md">here</a> (5000+) Learners!Most teams building AI products today know about evals and LLM judges, but they’re deeply confused about how to actually turn them into actionable improvement signals post production. Just throwing in LLM judges won’t magically fix your product. 🚨 You can also upgrade to our 1-year <a target="_blank" rel="noopener noreferrer nofollow" href="https://levelup-labs.ai/ai-builder-pass">AI Builder Pass</a>, which includes unlimited retakes, access to all our other courses, live sessions and partner credits from companies like Vapi AI, Arize, Cursor, Vercel, OpenAI, and more! Register for this course, and we’ll send you upgrade details over email.<ul><li>This course shows how to use offline evals, online evals, and production monitoring to drive continuous improvement, based on real production experience.</li><li>This course is for people who’ve already built AI systems and are stuck with evals that are noisy, expensive, or uselessIf you need something more foundational, check out our top-rated flagship<a target="_blank" rel="noopener noreferrer nofollow" href="https://maven.com/aishwarya-kiriti/genai-system-design"> enterprise AI course</a> taken by over 1500+ builders and leaders.</li><li>We follow a flipped-classroom format with async lecture content and meet twice a week for office hour style sessions to optimize two-way/interactive live time.</li></ul>

Move beyond one-time evals to continuous improvement flywheels for AI products

Understand what evals actually mean and why it’s misunderstood

Build a pre-deployment evaluation foundation from scratch

Choose the right types of metrics for the system you’re building

Monitor in production and close the improvement loop

You should have built small AI applications before and be familiar with concepts like RAG, MCP, and tool use.

A good understanding of AI system design

The course includes optional Python assignments, you're free to use coding agents, but basic coding skills are recommended

Working knowledge of Python/Coding (Optional)

Software/AI Engineers, Strategists, Data Professionals, Solution Architects and Consultants looking for systematic evaluation practices

Business Leaders and Product Managers seeking to gain the technical understanding of AI Product Lifecycle and intentional iteration

Founders &amp; teams stuck at systematic evaluations and monitoring, unsure how to turn signals into real product improvements.

The AI Evaluation Handbook

Agentic AI

Coding with AI

AI Workflows

Claude Code

OpenClaw

Vibe Coding

AI Evals

AI Transformation

RAG & Search

AI for PMs

AI for Engineers

AI for Designers

AI for Marketers

AI for Founders

Product Sense

Product Discovery

User Research

Prototyping

Growth

Analytics

Tech Foundations

Strategy

Influence

Leadership

Career Growth

Machine Learning

LLM Ops

Context Eng

Security

System Design

Figma

Design Systems

Visual Design

Design Strategy

Automation

Content Marketing

Demand Gen

Go-to-Market

Product Marketing

Positioning

Social Media

Brand

B2B Marketing

SEO & AEO

AI for Leaders

AI Governance

Communication

Management

People Operations

Exec Presence

Storytelling

Goal-setting

Personal Brand

Fundraising

Everyone

Operators

Data Scientists

Business Analysts

User Researchers

Customer Success

Project Managers

HR Professionals

Sales People

Lawyers

Finance

Investors

Real Estate

Educators

Creators

Welcome and Agenda Overview

What Are Evaluations and Why Are They Necessary?

Applying Evaluations in RAG and Agent Applications

Categorizing Evaluation Techniques: Programmatic vs. Manual

A Framework for Choosing Evaluations: Reliability, Scalability, Cost, and Value

Techniques for Improving LLM Judge Reliability

Auto Blocks Platform Demo

Q&A and Closing Remarks

Engineers and engineering leaders who build, deploy, and scale software systems.

Data scientists who apply statistical and ML techniques to extract insights from data.

Measuring and evaluating AI model performance, quality, and reliability for production use cases.

Applying AI tools and techniques to professional workflows — assign alongside more specific AI topics.

Master Evaluation Techniques for LLM Apps

What you'll learn

Why this topic matters

You'll learn from

Haroon Choudery