Stop Making LLMs Write Answers, Let Them Score

Many LLM requests do not require generating new text.

In routing, classification, policy checks, and ranking, the valid answers are already determined before the request is sent. There is only one real question: which answer best matches the input.

Take a support ticket as an example: 'I was charged twice for the same subscription.' The application needs to route it to billing, technical support, or account access.

Normal LLM decoding goes the whole way: the model generates a sentence token by token, such as 'Based on the customer query, this ticket should be forwarded to the billing department.' The application then parses the text to extract the decision. It goes around in a circle just to arrive at an answer that was already clear.

Structured output is better, but underneath it is still generation. The schema masks illegal tokens, so the model cannot write arbitrary content, but every brace, field name, and value in the JSON still pops out one token at a time.

Jev-style scoring takes a different approach. This is the inference pattern used by TypeSafe AI's Jev decision engine. Before inference, the application hands the query, state, and list of allowed answers directly to the model. The model does no text generation; it directly scores the predefined candidate answers:

billing → 0.90
technical support → 0.10
account access → 0.00

The application directly receives the labels and probability distribution.

The principle is not complicated. In the first step of generation, the model produces a vector whose dimension equals the vocabulary size for the next position, where each value is the logit of the corresponding token. Normal generation samples one token from this vector, appends it to the sequence, and repeats. But if you only care about a few predefined candidates, you only need to read out the logits corresponding to those candidates, ignore the other values, and apply softmax to the selected few.

Candidate labels must be single tokens. A visible word is not necessarily a single token: 'billing' might be one token in one tokenizer and two in another; 'technical support' definitely spans multiple positions. Use A, B, C as labels, describe the semantics in the prompt, and when the model processes the prompt it understands the semantics; we only need to check the logit at the last position.

There is a pitfall: tokenizers often encode leading spaces into tokens, so 'A' and ' A' may be two different token IDs. The chat template may also insert whitespace or control tokens before the answer position. Use the model's chat template to render the full prompt, send the expected continuation to /tokenize, and confirm that it is exactly one token.

Avi Chawla reproduced this inference path locally with SGLang in this article. The /v1/score endpoint accepts a prompt and label_token_ids, and returns normalized probabilities in a single forward pass. He specifically notes that what is reproduced is the inference path, not the complete Jev system; Jev also includes training and calibration work.

A commenter shared jevmlx, a local decision engine that runs on Apple Silicon. It compiles the schema into a prompt, scores all candidates for each field in a single forward pass, and assembles the JSON directly from the winners, with no text generation at all. On TypeSafe's public benchmark, Qwen3-8B with the labels scorer achieves 75% accuracy, higher than the 69.3% of generation-plus-parsing, with 0.6 seconds per item versus 1.6 seconds.

A commenter posted a benchmark scatter plot that places Jev and a bunch of alternatives in a speed-versus-accuracy coordinate system. A green dot named litjey-27b stands out in the upper left corner.

There are a few pitfalls to watch out for.

A developer pointed out that label-set drift is a production hazard: adding one valid answer invalidates all thresholds calibrated on the old answer set. The original author also admits that these probabilities are conditioned on the candidate set; add one label and all other probabilities change. The actual deployment unit is the model, prompt, tokenizer, label set, and thresholds—none can be missing.

The meaning of the probabilities is also easy to misread. 0.91 means billing gets 91% of the probability mass among the three candidates; it does not mean the model has 91% accuracy. Accuracy must be measured with labeled samples.

There is also the escape hatch. If a safety incident enters a router that only offers billing, technical support, and account access, the constrained softmax will assign all probability to the three wrong answers. Add an OTHER or ESCALATE to give 'none of the above' a way out.

A commenter summarized three system-level advantages of scoring over autoregressive generation:

1. Latency is only prefill, with zero decode steps. Autoregressive decoding is limited by memory bandwidth: every generated token requires reading the entire model weights. Scoring is a single forward pass: compute the prompt logits, slice out the target token IDs, and normalize. 300 ms can drop to 8 ms.

2. Zero KV cache allocation and pollution. Generating tokens requires allocating paged KV cache blocks, tracking, eviction, and cross-worker synchronization. A single scoring pass discards intermediate activations, keeps shared prefix caching intact, and is friendly to concurrent queries.

3. Confidence is an exact normalized probability with no sampling noise. Threshold routing becomes simple and deterministic: if max_p is below 0.85, escalate to a heavier reasoning model.

The applicable scenarios are clearly divided:

• LLM decoding: when new text is needed. It can also return labels, but requires generation followed by parsing.
• Structured output: when values are unknown but the schema is fixed. The model still generates token by token.
• Jev-style scoring: multiple-choice problems where the answer is known. It cannot write explanations or generate unknown field values.

Generation is not the only way for LLMs to reason. When the output set is known, the first next-token vector already contains the ranking. Return this ranking and bypass the autoregressive loop that the caller does not need.