Starcast
Join the waitlist
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
How to Find the Agent Failures Your Evals Miss with Scott Clark - #767
Coming soon to iPhone. Already have it? “Listen in Starcast” takes you straight there.
On this episode
About this episode
In this episode, Scott Clark, co-founder and CEO of Distributional, joins us to explore how teams can reliably operate and improve complex LLM systems and agents in production. Scott introduces a Maslow’s hierarchy of observability: telemetry for logging, monitoring for known signals, and post-production or online analytics to surface unknown unknowns. We dig into examples of real-world failures Scott’s team has seen in production systems, such as “lazy” tool-use hallucinations that standard evals miss, and how mapping traces into vector fingerprints enables clustering and topic discovery to uncover emergent behaviors. Scott explains how analytics can feed the data flywheel by generating…