Handwritten Agent Frameworks Have Reached Their Limit? Paper: Write the Exam First, Then Write the Code
The paper is titled “The End of Harness Engineering”—hand-writing the program around the model, as a skill, has reached its limit. The only remaining job for humans is to write the exam questions. This article combines Hanako’s long engineering post and the online debate to break down the logic and flaws behind this self-scoring framework.
A paper from a Chinese research team has a title that makes anyone building Agent frameworks uneasy: “The End of Harness Engineering.”
The argument can be compressed into one sentence: hand-writing the program around the model, as a skill, has reached its limit.
The so-called harness is the layer of program outside the model: responsible for calling the model, feeding it input, connecting tools, and receiving output. Today, most harnesses are written by hand by programmers. Hanako offered an analogy: this kind of manual work is “someone’s guess about a task on a certain afternoon.” After it is shipped, no one reads it for months, and no one scores it.
A harness written by a loop is different. It is a file that scores itself. A small model with only 4 billion parameters reads the failure logs and proposes 8 modifications; only the modification with the highest score on the held-out question set (validation questions not seen during training) survives. The large model stays frozen the whole time. Put a model like Claude Fable 5.2 in this position, and every round it rewrites the surrounding harness.
As a result, human work shrinks to one thing: the scoring set—the question set used to score the harness. The framework becomes disposable, and the exam becomes the work. Hanako compressed the entire paper into one sentence: write the exam first, then write the code.
Hanako already unpacked this “writing the harness through a loop” idea in the August long post “Loops and Graphs”. The core of that article was: how to stop staring at every step of the agent and only approve the final step.
Several key points:
- Loops live inside nodes; graphs live between nodes. Inside a node, it is “generate → check → correct → until green”; between nodes, it is splitting, parallelizing, merging, gating, and rolling back.
- Check conditions must be programmatically evaluable. “The test suite exit code is 0,” “every conclusion comes with a source line number,” and “the diff only touches the files listed in the plan” count as checks; “looks good” and “the model is confident” do not.
- Gating looks at blast radius, not confidence. Reversible and constrained channels open first; reversible but broad channels are guarded by deterministic checks; hard-to-reverse channels are shut down directly, no matter how high the score.
- Learning edges turn accepted results into constraints and send them back to the splitter, affecting the next split. A system without learning edges is merely fast; it does not become smarter.
Hanako turned this engineering method into a free course; the long post contains the full template and sequence.
Once the paper came out, the comments section had its own interpretations.
Some commenters said, so this job has now become writing evaluations—actually it has always been that way.
Some questioned it from a technical angle: if only the best of 8 edits survives each round, the held-out question set will gradually become a training set unless it is rotated regularly. This concern is concrete—the self-scoring loop may eventually overfit its own exam questions.
Some poured cold water on it from a business-model perspective: this mechanism will make model providers more money from compute. The analogy is hiring a contractor to renovate your house and letting them decide how much to charge, with you only being notified when the work is done.
Some also pointed out that this has long been the premise of the Nodarama project, not mad science. What is really worth discussing is whether, before continuing to pile up data centers, we should leave a little room for efficiency.
Regardless of whether the title is exaggerated, one trend is worth noting: people who write code may need to become people who write exam questions. Whoever owns the scoring set owns the output. Today’s harness engineers may need to start practicing how to produce a good set of exam questions.
发布时间: 2026-11-20 20:57