How to Use Decision Models? First Check Whether the Probabilities Are Accurate

Decision models have been discussed a lot recently, but the real question is: when should we let them make choices? Joe Barrow posted an article on X specifically explaining how Perplexity Decider works and the boundaries of using decision models. The original includes several Excalidraw diagrams, which are best viewed alongside it: Joe Barrow's article.

Decision models do something very narrow: given an input, choose among candidate actions. For ordinary users, think of it as a triage desk; for developers, it is a routing layer. It is not responsible for writing the full answer; it is responsible for sending the request to a more suitable model, tool, or process.

The problem is that routing often relies on confidence. If a decision model says it is 80% confident but is actually right only 60% of the time, routing by confidence will send errors into the fast lane. Latent Node put it directly in a reply: routing only by a decision model's confidence assumes probability calibration. Typed Decisions therefore scores calibration and accuracy together. One set of data it gives is: pplx-decider-v1.1-27b on 2,000 zero-shot decisions has accuracy 0.753 and KL 0.441. The dataset is on Hugging Face.

Another line of discussion is Typesafe AI. George McCarthy asked whether Typesafe AI still has a "secret sauce" beyond what open-source models can already reproduce. Among outside reactions, some joked about "2 years in stealth," while the founder of Jev implied that what they have is more complex than open-source models that appear overnight. Joe Barrow's reply was restrained: not sure, but training methods like RLCD, which uses reinforcement learning for calibrated decisions, and the data pipeline, might be interesting. This judgment is not settled, and there is no need to rush to take sides.

Joe also mentioned that work by @antoine_chaffin and others is well worth reading, and one can see traces of Ettin and early encoder work. This kind of context is more useful than looking at a single model name, because at least you can know where the method comes from.

The takeaway is not complicated. The value of decision models lies in routing, and the value of routing lies in trustworthy probabilities. If accuracy looks good but calibration is poor, the decision model becomes an error allocator that looks smart. In engineering, test calibration first, then talk about whether to integrate it. For ordinary users, whether decision models feel seamless depends on whether the system leaves uncertain cases to humans or more expensive models. For developers, benchmarks like Typed Decisions are more useful than just looking at demos, because at least they put "whether it is confident in the right way" on the table.