Don’t Let the Agent Act Yet: IBM Adds a Gate Before Tool Calls

IBM has published a paper on a decision gate for agents. The approach is not complicated: add a runtime check between the agent and the tools, without changing either the model or the tools.

This system is called JevAct. It first has the LLM convert domain policies into executable code checks and yes/no questions. Before each selected tool call is actually executed, the code checks the facts in the tool’s records, and a Jev-style judge model answers questions tied to the dialogue context. The judge model only performs prefill and outputs a probability judgment; it does not generate sentences. If a call fails the check, it is sent back to the agent with feedback so the agent can revise it within a limited scope.

The standout result is τ²-bench Telecom. Opus 4.6’s success rate rose from 39.5% to 80.7% across three runs. In the Telecom scenario, eight agents improved their average success rate by 16.9 percentage points across three runs. On BFCL, all nine agents improved their three-run success rates, while the 'at least one success' results were mixed. Airline showed small changes, varying by agent. The paper’s point is that the gains come from the harness.

In the comments, someone asked who checks the code after policies are converted into code. Others asked how much latency increases when every tool call goes through the gate; the source material gives no numbers. One commenter suggested that the limited Airline gains and clear Telecom gains mean the gate’s effectiveness depends on whether domain policies can be cleanly compiled into checks, and that Telecom’s policies are probably closer to rule-shaped. This explanation is reasonable, but it is still a commenter’s judgment, not a conclusion of the paper.

The paper also states its limitations. The Telecom gain did not appear after switching to the gpt-oss-120b user simulator. Among 260 labeled development-set write calls, 42 of 101 erroneous calls did not trigger any compiled check. The gate has blind spots, and those blind spots come from the compilation quality from policy to checks.

Another worthwhile perspective comes from the comments: change one policy rule, regenerate the gate, replay the same batch of tool traces, and see which decisions flip. This can reveal whether the gate is stricter than the policy itself. For people building agent systems, this is closer to real engineering practice than just looking at a benchmark score once.

Paper link: https://arxiv.org/abs/2610.38620

Chat with Paper: https://academy.dair.ai/papers/think-before-you-act-decision-gate-for-llm-agents-2610.38620