Agent Didn't Crash—It Just Consistently Answers Wrong

Agent observability is often described as installing a Datadog for agents. Harrison Chase doesn't see it that way. He wrote in a tweet that when people think of agent observability, they might think of Datadog for agents, but it's actually more like Databricks for agents. The original wording was short, and he added: some work has already been done in this direction, and more will be seen soon.

This analogy is worth unpacking. Datadog's stack solves whether a service is down, whether latency has spiked, whether error rates are abnormal. Switch to agents, and the dashboard will also tell you the process is alive, token consumption is normal, and tool-call success rate is good. Then the customer sends a screenshot: the answer is wrong, and wrong with great confidence.

In a reply, Luis Junior put this layer very directly: Datadog will alert you that the agent crashed. The trouble is the agent hasn't crashed; it just consistently answers wrong. You can't see it on the dashboard, but you can see it in customer complaints. He also asked whether agent observability is fundamentally an infrastructure problem, or whether someone finally has to define what a correct answer means.

This question is not trivial. An agent's output is not fixed JSON, nor a 200 status code. It may call tools, read documents, perform multi-step reasoning, and finally produce a plausible-sounding paragraph. Monitoring can catch timeouts and exceptions; it cannot catch semantic errors. Harrison's Databricks analogy is more like saying that agent observability doesn't only watch runtime; it also has to handle the dirty work of data, evaluation, experimentation, and feedback. Databricks builds infrastructure for data and AI workflows, not just an alerting dashboard. For agents to iterate, you need to know which batch of inputs failed, how failed samples flow back, how an evaluation set grows, and whether version changes reintroduced old problems.

The irony is that many teams first make traces look beautiful, then discover the most critical field is the correct answer. Without evaluation, observability easily becomes an expensive dashboard. With evaluation, there is a chance to turn customer complaints into the next round of test cases. Harrison says some work has already been done in this direction, and more is coming. What's really worth watching is not a few more dashboards, but whether agent errors can be located, reproduced, and written back into the system.

Original tweet: Harrison Chase