Shreya Rajpal

Improve reliability of your AI applications

Apple

University of Illinois

IIT Delhi

CEO and Cofounder, Guardrails AI

LLM applications require fundamentally new abstractions that systematically validate their performance

Why validation of AI applications is needed

Explore common techniques used to improve reliability for AI applications and evaluate which technique to adopt

Survey of AI reliability techniques

Hands-on demo of building an AI validator for a specific use case

How to build robust AI validators

Learn proven approaches for quickly improving AI applications.  Build AI that works better than the competition, regardless of the use-case.

ML Engineer with 20 years of experience

Airbnb

GitHub

DataRobot

AlixPartners

Hamel Husain

ML Engineer with 20 years of experience.

ML Systems & Applied AI Evals Researcher

Google

UC Berkeley

Stanford University

Shreya Shankar

ML Systems Researcher Making AI Evaluation Work in Practice

Cohort 5

Cohort 6

🚨 September 2026 cohort's material is completely refreshed. 🚨All students get:<ul><li>♾️ Unlimited access to future cohorts &amp; office hours: never worry about timing or missing new material.</li><li>🗄️ Lifetime access to all materials!</li><li>🤖 6 months of unlimited access to our new AI Eval Assistant (more info below).</li><li>🧑‍🏫 10+ hours of office hours to maximize the value of live interaction.</li><li>🏫 A Discord community with continuous access to instructors to get unstuck (even after the course!).</li></ul>---Do you catch yourself asking any of the following questions while building AI applications?1. How do I test applications when the outputs require subjective judgements?2. If I change the prompt, how do I know I'm not breaking something else?3. Where should I focus my engineering efforts? Do I need to test everything?4. What if I have no data or customers, where do I start?5. What metrics should I track? What tools should I use?6. Can I automate testing and evaluation? If so, how do I trust it?If so, this course is for you.All sessions are live and recorded.

Stop guessing if your AI works. Build the feedback loops that make it better.

How To Collect Data For Evals

Get immediate clarity and direction with Error Analysis

Implement Effective Evaluations

Master Architecture-Specific Eval Strategies

Run Evals In Production

Ensure That Evals Lead To High ROI

Engineers and PMs who ship prompt changes and hope nothing breaks. (You'll learn to measure impact before and after every change.)

Teams still spot-checking AI outputs by hand instead of measuring systematically. (You'll learn how build automated evals you can trust.)

Leaders who don't know where their AI is failing or where to invest resources. You'll learn how to systematically find &amp; prioritize issues.

Learn practical ways to lead in responsible AI, with tools you can apply today. 

Responsible AI Advisor & Instructor 

Ahold Delhaize

Johnson & Johnson

ABN AMRO

Hinda Haned

Legatus I

Legatus II

Responsible AI is no longer a niche topic, it’s on the agenda of regulators, boards, and business leaders everywhere. In 2024 alone, governments worldwide issued more than twice as many AI-related regulations as the year before. Add to that more than 80 frameworks, countless checklists, and a steady stream of new tools, and the field quickly feels overwhelming. The real challenge isn’t whether AI should be used responsibly; it’s knowing where to begin. 🎯This course brings you back to the foundations: cutting through the noise, grounding you in what matters, and giving you a clear path forward.Here is detailed breakout of the course:🟩 Week 1: Power Session | Responsible AI PrimerA focused 3-hour session that cuts through the noise and gives you a practical overview of Responsible AI governance.🟩 Week 2: Mentoring &amp; Application | RAI Lab<blockquote>In the second week, you’ll have up to three hours of 1-to-1 mentoring to apply the tools directly to your own context. This may include building a governance checklist, evaluating a vendor, or stress-testing a project plan. The goal is for you to walk away with a concrete, business-ready action plan tailored to your organisation.</blockquote>

Your Responsible AI power session: from principles to practice

Understand core Responsible AI principles in practical business contexts

Recognise common AI risks through real-world system failures

Use a simple Responsible AI risk assessment framework

Define actionable first steps toward Responsible AI adoption

Anyone seeking a clear, practical intro to RAI with real-world examples to apply best practices or guide others effectively.

Leaders seeking to build or improve RAI governance with concrete SOPs and case studies from leading organizations.

Professionals who need to understand and apply RAI to meet regulatory and ethical standards, using field-tested examples.

Learn to achieve reliable AI outcomes without prompt packs, system prompts, or agent tricks—use structured inputs for consistent results.

Creator, AIgogy Framework™ | Advisor

Rockford University

Silali Banerjee

Creator of AIgogy Framework™ for reliable, structured AI outcomes

July2026Cohort

Pretty much anyone using an AI chatbot has gone through the frustration of a response that isn't exactly what they wanted. Problems include hallucinated, drifted, generic, and inconsistent output. Sometimes all of these happen; other times, one or more.People try prompt improvisation, prompt tweaking, prompt packs, prompt libraries, trial and error, tries and retries. Sometimes it works; other times, it does not.Too many details in a prompt cause prompt bloating and do not solve the problem. Adding context helps, but too much context creates problems.AI agents and skills do not solve this problem with open-ended work either.People give up, thinking LLMs are probabilistic and cannot work better.That is not true.It may never be as deterministic as systems in terms of consistency and reliability, but it can be mostly consistent and reliable.Similarly, users can create a path with an LLM using the right framework. Users can guide their request through the LLM toward their desired outcome by using the right framework.Learn to use this science-backed, proven framework.It is based on research (<a target="_blank" rel="noopener noreferrer nofollow" href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6516618">https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6516618</a>) and has been tested in practice.<hr>

User's framework for moving beyond trial-and-error prompting with any LLM.

Achieve More Reliable AI Outcomes

Reflect Your Voice and Working Style

Embed Your Judgment and Standards

Reduce Rework and Increase Leverage

Leaders (Decision Owners) Leaders accountable for outcomes who need AI to reflect their judgment without reviewing or correcting every output

Operators (Execution Owners) Operators responsible for consistent delivery who need AI to produce aligned outputs without repeated rework

Knowledge Workers (Expertise Carriers) Experts who need AI to apply their domain knowledge and standards, not generate generic outputs.

 In one day, validate YOUR AI Opportunity & leave with a 90-day plan.

ICF MCC · MBA ESSEC · 6,500+ sessions 

Judith Glory

ESSEC MBA · ICF MCC · Executive Coach · 6,500+ sessions

Stop debating about AI. In one day, validate your opportunity and leave with a 90-day plan.

Cohort 2

The pressure to act on AI is real. But most leaders are making decisions based on hype, vendor pressure, or fear of being left behind — not on evidence.The result is predictable: pilots that fade, budgets wasted on tools that never get used, and leadership teams that keep debating instead of deciding.This workshop exists because there is a better way. In one structured day, you apply a proven framework to your team's specific reality — not AI in general, not someone else's use case. You evaluate what you do today. You identify where AI could genuinely help. You name every risk and price it. And you make a clear go or no-go call you can stand behind.Not every team should move forward with AI right now. If your numbers say no, this workshop will tell you that clearly — and give you the conditions under which yes becomes the right answer. A No-Go decision backed by data is worth more than a yes driven by pressure.

Managers and leaders navigating the AI moment who need clarity, not inspiration — and a plan they can defend to senior leadership team.

Team managers, VPs, and senior leaders who sense the AI opportunity but need evidence to act — or to confidently say not yet.

Leaders responsible for team output who are ready to validate their AI opportunity once and for all — no tech background needed.

The AI Evaluation Handbook

Agentic AI

Coding with AI

AI Workflows

Claude Code

OpenClaw

Vibe Coding

AI Evals

AI Transformation

RAG & Search

AI for PMs

AI for Engineers

AI for Designers

AI for Marketers

AI for Founders

Product Sense

Product Discovery

User Research

Prototyping

Growth

Analytics

Tech Foundations

Strategy

Influence

Leadership

Career Growth

Machine Learning

LLM Ops

Context Eng

Security

System Design

Figma

Design Systems

Visual Design

Design Strategy

Automation

Content Marketing

Demand Gen

Go-to-Market

Product Marketing

Positioning

Social Media

Brand

B2B Marketing

SEO & AEO

AI for Leaders

AI Governance

Communication

Management

People Operations

Exec Presence

Storytelling

Goal-setting

Personal Brand

Fundraising

Everyone

Operators

Data Scientists

Business Analysts

User Researchers

Customer Success

Project Managers

HR Professionals

Sales People

Lawyers

Finance

Investors

Real Estate

Educators

Creators

Introduction and Housekeeping

The Importance of AI Application Reliability

Speaker's Background: A Decade in Machine Learning

Real-World Examples of AI Unreliability

AI Reliability vs. Traditional Software Reliability

Defining AI Reliability: Constraining Models for Specific Tasks

Survey of AI Reliability Methodologies and Evaluation Criteria

Methodology 1: Retrieval-Augmented Generation (RAG)

Methodology 2: Fine-Tuning

Methodology 3: Input and Output Verification (Guardrailing)

Deep Dive: Guardrails Hub and Practical Examples

Live Demo: Building a PII Detection Guardrail

Audience Q&A Session

Conclusion

Engineers and engineering leaders who build, deploy, and scale software systems.

Measuring and evaluating AI model performance, quality, and reliability for production use cases.

Applying AI tools and techniques to professional workflows — assign alongside more specific AI topics.

Operating and deploying large language models in production — fine-tuning, inference, monitoring, and cost optimization.

Improve reliability of your AI applications

What you'll learn

Why this topic matters

You'll learn from

Shreya Rajpal