Every New Message Triggers a Full Re-read — Sage2 Wants to Delete That Step

Real-time conversational products carry an inconspicuous cost: with every new message, the model has to re-read the entire transcript. The more turns there are, the harder it becomes to suppress latency and cost. Sage2, released by Levanto Labs, splits extraction and decision-making into two models running on the same live stream — with the explicit goal of eliminating that re-read.

Avi Chawla provides the background in his tweet. The previous generation, Jev, answered fixed questions over fixed inputs; dropped into a live chat, it had to re-read the full transcript on every update. Sage2 changes the structure: the extraction model turns each new message into structured fields in about 150ms, with each field bound to the source span it came from; those fields become the decision options. The decision model scores as it reads the text stream, returning typed answers with probabilities. The whole pipeline goes from raw text to action in about 200ms, without ever re-reading the transcript. Sage2 is built by Levanto Labs, which says it previously built the first decision model.

The video shows a demo of Sage2 in a broadband customer-support chat, produced by Avi Chawla together with the team.

In its launch tweet, Levanto Labs cites two numbers: the extraction model is 166× cheaper than Opus 5.5, and the decision model is 3× faster because it processes context as it reads. Such figures only mean something with a clearly stated methodology — especially the 166× cheaper claim, which depends on the task, input length, and the version being compared. The launch tweet also mentions a 20M-token free tier, plus near_ai and an agent-facing mpp integration.

The extraction model is new, and the team says it couldn't find an existing benchmark — so it built Extract Bench. It spans 12 public datasets, with v0.1 containing 1,844 documents covering invoices, receipts, forms, contracts, reports, emails, and Q&A. Scoring looks at field accuracy, and also at whether the model correctly returns null when a field is absent from the document.

On the leaderboard, Sage2 scores 77.8 on field accuracy and 72.2 on correct rejection of absent fields. NuExtract3 comes in at 76.1 and 57.0, Clairic at 75.3 and 59.0, Schematron-3B at 59.9 and 54.3, and GLiNER2.5 at 35.7 and 44.1.

This leaderboard is maintained by Levanto, which also sells Sage2 — a fact worth factoring in when reading it. Sage2's abstention threshold was also tuned on the fit half. Fortunately, the benchmark is open source, with predictions, reproduction scripts, and CI re-scoring all available on GitHub.

The comments are more worth reading than the launch copy. One person asks how Sage2 handles a user who changes their mind mid-conversation or contradicts something they said earlier. Another follows up: if a decision requires reasoning about something the extractor discarded, and Sage2 only stores post-extraction facts, that context can't be recovered without re-reading the transcript. Someone else asks: given that structured fields already exist, why not just write logic rules instead of adding yet another decision model? Another, more engineering-flavored question is more concrete: when do intermediate scores trigger a second check?

The launch video can't answer these; you'd have to look at how the actual system is designed. What's worth noting about Sage2 is that it breaks state updates in real-time conversation down into testable engineering primitives: extraction has fields and source spans, decision-making has probabilities and latency. The expensive part of real-time systems is usually how you fix things after they go wrong — and a 77.8 on a benchmark doesn't help with that part.

Sources: Avi Chawla's tweet, Levanto Labs' launch tweet, Extract Bench GitHub.