The Same GPT-5.5, 234 Findings vs. 74: The Code Review Gap May Not Be in the Model
Javi compared GPT-5.5 on the same set of 200 real PRs in a review harness versus Codex, matching expert comments 234 to 74. Alibaba's OpenCodeReview paper reports the same row of data, while also cautioning that this is their own harness running their own benchmark. The model didn't change; the scaffolding did, and the scores differ by a wide margin.
Javi posted a comparison on X: the same GPT-5.5, the same 200 real PRs, matched 234 expert comments in one review harness, but only 74 when placed in Codex. He asks: is the problem the model or the harness?
Original tweet. Javi's reply provided the paper and table: the same row of data is 21.0% vs. 8.4% SEM-F1, and 422K vs. 525K tokens. AACR-Bench contains 200 real PRs, 10 languages, and 1,505 expert-verified comments.

The paper is titled OpenCodeReview: Determinism over Non-Determinism for Cost-Effective Agent-Based Code Review, with authors mainly from Alibaba, Nanjing University, and others; the code is open on GitHub. What it changes is the review process outside the model. The model stays unchanged.
For the GPT-5.5 row: OpenCodeReview has SEM-F1 21.00%, precision 32.10%, recall 15.50%, and generated 728 comments, of which 234 matched; Codex has SEM-F1 8.36%, precision 27.82%, recall 4.92%, and generated 266 comments, of which 74 matched. Token consumption is close between the two systems: OpenCodeReview takes 2m51s, Codex 2m58s. The number of matches differs by more than 3x, and SEM-F1 differs by 2.5x.
The paper breaks the harness into three stages. Rule-guided dispatch first decides which files to review and which rules to review them with, avoiding the agent picking files on its own every run. File-level review assigns a SubAgent to each file, exposes only six tools, and caps output: read file, find file, search code, view diffs of other files, write comment, and finish. Independent reflection uses only the diff to filter comments and perform falsification; it does not generate new comments. Its goal is to remove hallucinated comments directly contradicted by the diff while not mistakenly removing evidence the agent found in the repository.

These design choices point to one judgment: the bottleneck for code review agents is often not whether the model can write comments, but how information comes in and how comments are filtered out. Giving the agent maximum freedom may buy a longer tool chain, a larger token bill, and less stable results. The paper's title states it in reverse—determinism over non-determinism—using deterministic engineering to contain uncertainty.

But don't treat this single row of data as the final word. Javi also cautions that this is Alibaba's own harness running its own benchmark, so it can only serve as one data point. Semantic matching in AACR-Bench is judged by an LLM judge, and the paper says each configuration is run five times and averaged. None of this can replace review quality in real teams. A more robust reading is: model capability still matters—the six backends score from 17.90% to 25.10%—but the same GPT-5.5 changing harness can go from 8.36 to 21.00, so the scaffolding can account for at least a large part of the gap.
So the most interesting thing about the 234-to-74 numbers is that they pull the question back from model parameters to engineering constraints. It's easiest to blame the model; changing the harness is harder. Next time a review agent misses comments, first look at what context it received, what tools it can call, and whether comments were filtered. The model didn't change, but the score did; the answer usually hides in these three layers.
发布时间: 2026-11-18 02:31