720,000 AI decision records released, but before you use them, read this comment in the replies

Maziyar Panahi posted on X saying he had released 5 datasets and 720,000 decision records, all public. The use cases listed in the tweet are very specific: tool calling, agent traces, customer service, routing, and policy checks. Every record comes with a source label and Jev's answer, and can be used to train your own small decision-making model—even run it internally in a hospital.

720,000 sounds like a lot, but it becomes clearer when broken down. In replies, he expanded on the details of the three main datasets.

Policy Verification 180K, 180,000 rows, 45,281 decision groups, 84,007 unique upstream source rows. The task is to judge whether a response follows a given principle, which one is better, and whether it has gone off track. It is suitable for reward models, response ranking, and calibration. Jev consumed 220 million input tokens, output 4.38 million tokens, total cost $9.30, zero errors.

Intent and Domain Routing 130K, 130,000 rows, 129,976 decision groups. Questions to answer include: what the user wants to do, which domain it belongs to, what kind of RAG task it is, and what kind of agent workload it is. Suitable for training intent classifiers, model routers, and multilingual routing. Jev consumed 240 million input tokens, cost $10.20, zero errors.

Agent Trace Audit 100K, 100,000 rows, but only 5,556 unique upstream source rows. Each source row was split into multiple decisions: what was the outcome of this agent run, which tools were used, whether a patch was produced, and whether the next step should be auto-close or hand off to a human. Suitable for trace auditing, failure analysis, and reward shaping. Jev consumed 310 million input tokens, cost $13.01, zero errors.

The three datasets together cost less than $33, ran on about 770 million input tokens, and reported no errors. At this price, it lowers the barrier to "using a model for labeling" another notch.

But there is a set of numbers in the dataset cards more noteworthy than cost.

Jev's "descriptive consistency" with upstream source labels is as follows: Policy Verification 74.60%, Intent Domain Routing 92.52%, Agent Trace Audit 67.50%. The card states clearly that this is behavioral auditing, not an accuracy or benchmark claim, and that the upstream labels themselves may also be noisy.

The breakdown for Agent Trace Audit is more specific: trace_outcome consistency 26.62%, trace_disposition 36.53%, patch_presence 49.39%. In other words, when judging "did this agent run actually succeed," Jev and the external resolver disagree three-quarters of the time. This does not necessarily mean Jev is wrong; it more likely shows that defining agent success or failure using resolver results is itself very hard to align.

In the tweet's replies, Oleg Zankov's response was highly upvoted. He said: the rows where Jev and the source labels disagree may be especially useful; before distillation, he suggested sampling a batch and looking at them, to separate true label corrections from errors that should not be inherited by a local decision-making model. Maziyar replied: "This is the most requested feature here."

This exchange highlights the practical question of how this data should actually be used. With 720,000 decisions, if you directly use them as supervision signals to train a small model, you are effectively assuming every Jev output is correct. But the card explicitly says Jev's outputs are model predictions, not ground truth, and their confidence has not been calibrated. The disagreements are exactly where information density is highest; keeping them lets you study teacher-human disagreement, while throwing them away filters out the most interesting part.

There is another detail worth noting. The 100,000 rows in Agent Trace Audit were expanded from 5,556 upstream source rows, averaging 18 decision questions derived from each source row. This means the same trace segment was read repeatedly and questioned repeatedly; each row retains the full request, original response, token usage, and latency, and can be deduplicated by request_json or group_id. If someone directly splits the training set by row count, samples from the same group may appear in both train and test, and evaluation results will be inflated. The card says component-safe splits were used, with isolation by group and source row already applied, but you still need to pay attention when processing it yourself.

Jev itself is jev-1.13.0, and all five datasets were labeled with this version. The upstream sources for Policy Verification and Intent Domain Routing are mainly NVIDIA's HelpSteer2 and HelpSteer3; Agent Trace Audit comes from nvidia/Open-SWE-Traces, based on repository tasks from nebius/SWE-rebench-V2 under the MIT license.

In the tweet, Maziyar used the phrase "train your own tiny decider and keep it inside your hospital," meaning these decisions can be distilled into a small model, deployed locally, with data never leaving the premises. This direction is very practical for privacy-sensitive scenarios; hospitals, banks, and government agencies all have similar needs.

But back to Oleg's comment: before using it, look at the disagreements first. Among the 720,000 records, the parts where the model and source labels agree can be used directly as supervision signals; the parts where they disagree are exactly what needs a human look. If you skip this step before distillation, you package Jev's judgments and the noise in upstream labels together into a small model, and it will be hard to separate them later.

Dataset links can all be found on Hugging Face; searching the three main names will get you there: Jev-Policy-Verification-180K, Jev-Intent-Domain-Routing-130K, Jev-Agent-Trace-Audit-100K. The full list is in the Jevmaxxing collection.