Free Lesson
Create MCP Tool Evals Before You Ship
Part of The AI Evaluation Handbook
45 min
Jan 28, 2026 12:00 PM
Virtual (Zoom)
In this video
What you'll learn
Understand tool calls and failure modes
Learn how LLMs select MCP tools, what drives success rate, and common routing failures.
Set up MCPJam with a sample MCP server
Install MCPJam, connect to a remote MCP server, and verify tool calls end-to-end.
Build a repeatable tools eval harness
Add sample prompts + assertions to validate tool selection and outputs for key user requests.
Run evals and collect data
Execute the harness, read pass/fail results, and diagnose tool mismatches.
Next Steps: Operationalize evals to prevent drift
How to scale your workflow by with the MCPJam CLI and CI, collect real user prompts, and re-run as APIs and data evolve.
Why this topic matters
MCP makes LLM apps powerful, but one wrong tool call can break flows, waste tokens, or return bad data. Tool evals measure task success, catch selection mistakes early, and detect drift as APIs and data change. We'll learn how to set up a repeatable eval harness with MCPJam to improve reliability now and monitor regressions as you ship updates.
You'll learn from
Emmanuel Paraskakis
Founder, Level 250 | 3x VP, API Product
Previously at
