Your Agent Tests All Pass, but It Still Fails in Production? Google’s 95-Minute Course: The Process Went Off Track Long Ago

Your Agent tests all pass, but it still fails in production. It’s not that the model is bad—no one has looked at how it does the work.

Google recently released a 95-minute course, centered on ADK eval. Two built-in metrics: response match checks whether the final output is compliant, and trajectory checks the intermediate path the agent takes through the entire task—what it looked up, what it called, and how many times it changed its mind along the way. Both metrics can be plugged directly into pytest and CI/CD, so every prompt change automatically runs regression. The whole setup takes less than half an hour.

trajectory deserves its own mention. It exposes an often overlooked blind spot: the answer may be correct while the process is completely uncontrollable. The pitfalls that cause production failures often aren’t in the final result; the process went off track long ago.

The course also offers a counterintuitive solution: if the agent’s output is always noncompliant, don’t just compare it against a reference answer. Write a checker that verifies each constraint one by one, then calculate a score. ADK optimize runs 30 candidates per round, taking 8 to 10 minutes, and automatically revises the instruction. In plain terms, the agent grades its own homework.

But evaluators aren’t omnipotent. The course explicitly warns: if you use an LLM as a judge and choose the wrong model, running more rounds later is wasted effort. First validate with a few dozen manually labeled examples, then scale up.

After watching this course, one developer printed out all the intermediate steps of their agent and went through them, finding that several pitfalls weren’t wrong answers—the process had gone off track long ago. Chinese subtitles are already available. Save it and take time to go through it carefully.