91% vs. 45%, half the tokens, and the winner is that pile of notebooks nobody looks at

Databricks co-founder Ali Ghodsi posted on X that he is 10x better than he was a year ago, and the whole company now runs on Genie. He attributed the reason to Agentic Ontology, an enterprise knowledge layer that is continuously curated, evolved, and shared across the company. He did not mention how many models had been swapped.

This sounds like product marketing. But Databricks then released a set of tests done with Turing, and the numbers are pretty direct: on the same batch of enterprise data analytics tasks, Genie's pass rate was 91%, while two frontier agents were 45% and 58% respectively. Genie used an average of 430,000 tokens per run, versus 890,000 and 1.05 million for the competitors, and was 3 to 4 times faster.

To understand this gap, you first need to look at how the test was set up.

They fictionalized a UK energy company, Calderwick Energy, with hundreds of tables; the largest stores about 120 million rows of half-hourly electricity meter data. But what really makes the tasks hard is what has piled up on top of those tables: notebooks, scheduled jobs, dashboards, saved SQL, code, metric definitions, documents, and Genie Agents. They also deliberately left in the mess found in real environments: duplicated notebooks, abandoned analyses, old documents, and reports nobody has touched for months.

Traditional data agent benchmarks mostly provide only table schemas, and anyone can get roughly similar scores on them. In real companies, the correct answer often lies in a SQL comment someone wrote six months ago, and the table schema itself cannot explain it.

The blog gives a specific example. The question is: What is the FY24/25 planned rate for Calderwick Energy's commercial team?

In the billing cost table, there is a field cfd_rate_p, and the row for FY24/25 says 0.31 pence per kWh. Find the table, filter by date, return that number, and it looks done.

But the notebook that built this table says the commercial team sets a planned rate once a year, and this mart uses the FY25/26 assumption, applied backwards to earlier billing periods. The notebook also has a history table: FY24/25 is 0.52, FY25/26 is 0.31, and the FY26/27 draft is 0.45.

The question asks for the rate set by the commercial team for FY24/25, so the answer should be 0.52. The 0.31 in the table is an assumption that was changed later.

Genie Ontology extracted the historical rates from that notebook and linked them to the source. Genie answered 0.52 on the first try, and in all five runs. The other two agents: one kept answering 0.31, and the other answered 0.31 and then asked whether there was an annually set rate.

What this case shows is not complicated. The numbers in the table are not the same as the business definition. Definitions are scattered across notebooks, dashboards, and saved SQL, and someone needs to read them, maintain them, and turn them into something agents can find.

The blog puts it quite directly: the "intelligence" generated by tokens is being commoditized, and what really creates the gap is the context an agent can retrieve.

That is exactly what Genie Ontology does: it automatically extracts and organizes business knowledge from notebooks, dashboards, SQL, and documents in the workspace, and gives all agents a shared understanding. It is not about letting each agent rummage through files all over the place, and even then it may not find the right thing.

Of course, these numbers have their own preconditions. The test ran in a Databricks environment that had already accumulated a large amount of workspace assets, with data governed by Unity Catalog. For a deployment that has just gone live and has barely any notebooks written, or for questions that can be answered directly from the tables, the gap between agents would be much smaller. The blog admits this itself, and uses it to explain why scores are similar across other benchmarks.

In other words, the value of this system is proportional to the assets an enterprise has accumulated itself. The messier and thicker the assets, the more valuable organizing them becomes.

The last line of the blog: tokens are depreciating; context is appreciating. For people building agents, this is not new, but the 91% vs. 45% gap at least gives them evidence they can take into an argument.