Most People Using AI Are Still at Level One: What Exactly Is the 'Harness' Karpathy Talks About
A tweet resurfaced something Karpathy said before: don't rush to put agents to work; first build the harness around the model. Livsun breaks AI engineering into five levels. Most people stop at level one, and the real gap comes from the later levels.
A tweet resurfaced a passage from Andrej Karpathy: the biggest mistake in AI right now is forcing agents to do work without first mastering harness engineering around the model. The tweet also says OpenAI made this mistake in 2016 and paid a five-year price. The details of this paraphrase may not be fully accurate, but the direction is worth considering: many people focus on agents and ignore the model, context, and runtime environment underneath the agent.
The video attached to the tweet is here:
Livsun then wrote a long post dividing AI engineering into five levels: Prompt, Context, Harness, Loop, Graph. Each level answers the same question: what can the model actually get at each step. New models come every few weeks, but these five levels will not really expire, because each level wraps the next.
Level 1: Prompt engineering, you are the loop. Open a chat box, type a sentence, read the answer, revise the sentence. The model answers in seconds, the human revises in minutes, and every retry waits for a person. To tell whether you are at this level: your proudest trick is a magic phrase, such as "think step by step" or "you are a world-class expert," and you have a notes file full of templates. Magic phrases help, but one sentence cannot give the model facts it does not have, cannot call tools, and cannot check its own results.
Level 2: Context engineering, fill the window. In June 2025, Shopify CEO Tobi Lütke said he preferred the term context engineering because it better describes the core skill: giving the LLM enough context for the task to be solvable. Karpathy replied +1 and gave a definition that has since been quoted repeatedly: context engineering is the delicate art and science of "filling the context window with just the right information for the next step." The last two words are the point: the next step. The model does not know your project, your rules, or the config you broke yesterday. It only knows what is in the window at this moment. So stop polishing sentences and start budgeting the window: put unchanging rules in the system prompt and project files like CLAUDE.md; tool descriptions should let the model tell which one to choose; memory should hold prior decisions and state; retrieval should include only the relevant 3 files, not the 300 that exist; put the task itself last, after the earlier information has loaded. Bottleneck: context is everything the model can see, but it is still one call, and the window is still assembled by hand. To make the model run code, open files, check results, and try again, you need a machine. That machine is Level 3.
Level 3: Harness engineering, the model is the engine, the harness is the car. The harness is the ring around the model: the loop that feeds tool results back in, the tools themselves, permissions, sandboxes, memory files, event hooks, and traces. Claude Code is a harness, and so is Codex CLI. Put the same model into two harnesses and you get two different agents. According to Livsun's article, OpenAI published "Harness engineering: leveraging Codex in an agent-first world" in February, in which a repository grew to about one million lines in five months, with Codex writing code and engineers designing the world it works in. Researchers from Fudan University and Peking University gave a set of numbers in April: the paper Agentic Harness Engineering had one agent do only one thing—continuously modify another coding agent's harness and check whether each modification matched predictions. After ten rounds, Terminal-Bench 2 pass@1 rose from 69.7% to 77.0%, surpassing the human-designed Codex-CLI harness at 71.9%; the frozen harness used 12% fewer tokens on SWE-bench Verified; the gains came from tools, middleware, and long-term memory, with the system prompt contributing the least. The last sentence deserves a second look: the place where most people spend the most time may be the least important. The first harness action you can take today: give the agent a gate it cannot get around. In Claude Code, the Stop hook runs every time the agent wants to finish; exit code 2 blocks completion and sends the error back. That way "done" means the tests pass. The model thinking it is done does not count. Bottleneck: even the best harness is still waiting for you. Every run starts with a human entering the task. You are not in the loop during the run, but you are still the trigger.
Level 4: Loop engineering, stop being the person who writes prompts. On June 7, 2025, Addy Osmani named this level. His article "Loop Engineering: Designing loops that prompt coding agents" defines it in one sentence: loop engineering is replacing yourself with a loop that writes prompts to the agent. He puts this level above harness. People close to the tools get there first: Peter Steinberger said you should no longer write prompts for a coding agent; you should design the loop that prompts the agent. Boris Cherny said he no longer prompts Claude; his job now is writing loops. Osmani's loop anatomy has five parts plus a place to store state: automation that wakes up on a schedule to find work and triage; worktrees so parallel agents do not touch each other's files; skills that store project knowledge and load when needed; plugins and connectors that plug into existing tools; sub-agents that separate the proposer from the checker; and a state file or board that records what is done and what comes next. You can start tonight: Claude Code has /loop; give it an interval and a task and it will run repeatedly; without an interval, the model sets its own pace. The trap is also here. A basic agent loop has only one way to end: the model stops requesting tools. Put that on a timer and you have built a machine that can spend money all night. Every loop worth running needs brakes: a completion check outside the model, such as the Stop hook from Level 3; budgets for maximum runs, maximum tokens, and maximum spend; a dry-run rule that stops if two rounds produce nothing new; and state written at each step so it can resume after a crash instead of starting over. Osmani makes it clear that a loop is not full autonomy; checking the work and keeping judgment are still your job. Loops remove typing, not thinking. Bottleneck: a loop is still one track. One agent, one queue, one thing at a time, repeated. When the task is broad—for example, review 40 files, check 12 sources, audit every route—a loop just does it more slowly. The top layer needs to change shape.
Level 5: Graph engineering, draw the shape of the task. At this level, stop thinking in steps and think in nodes and edges. A node is a bounded task with clear inputs and outputs. An edge exists only when the output of one node feeds the input of another. Many agents described as "first step, second step, third step" are actually graphs stuffed with fake edges. Half the steps do not read each other's outputs at all; there is no reason for them to queue. Delete the fake edges and the line becomes a diamond: split, where one node breaks the task into independent chunks; fan out, where multiple agents work at the same time, each with clean context; verify, where a gate tries to kill each finding and only survivors pass; merge, where one node looks only at the survivors and writes the answer. This is no longer just theory. According to Livsun's article, Claude Code's dynamic workflows have been a research preview since June, letting Claude write the plan as a JavaScript script and run it with a swarm of subagents, dozens to hundreds in a single session. Routing is in code, so every run works the same way; the connections between agents are ordinary code, no longer a single model turn. Two rules make graphs fast and cheap: wait for all results only when a node truly needs all of them at once, such as deduplication and sorting; everywhere else, let each item flow directly to the next node. Layer the models: use cheap models for repetitive fan-out nodes, and the strongest model for judgment and merging.
The 30-second test: you repeatedly revise the same instruction until it works—Level 1. You load project files, the right docs, and examples before every task—Level 2. Your agent can run tools, has hooks, and cannot claim it is done if checks fail—Level 3. Work can start without you typing—Level 4. One request fans out to multiple agents, is verified, and merged back—Level 5.
Five things you can do this week, one per level: delete every "you are an expert" from your most-used prompts and replace it with one good example output. Write a one-page project file with rules, stack, decisions, and completed items; load it before every session and count how many revisions you save. Add the Level 3 Stop hook so the agent cannot finish when tests fail. Put a /loop on your todo list or inbox, with a budget and a state file; run it overnight and read the state at breakfast. Build one diamond: split a review across multiple files, fan out, have a skeptic agent attack each finding, and merge the survivors into one report.
Livsun's conclusion is direct: prompt writers write sentences; Level 5 engineers write the shape of the work, and the sentences write themselves. Models will keep changing: new Claude, new GPT, new Gemini, a new open-weight giant every two weeks. Each is only an engine. The layers you build around it are what will not expire on release day. Most people will keep polishing first-level sentences. A few will climb upward and run a fleet while others are still typing. The tweet ends by saying this is worth more than any $3,500 agentic engineering course. After seeing the five-level path, that sentence does not feel exaggerated.
Atish Raina also recorded a 30-minute AI Agentic Systems video explaining where tools, retrieval, memory, RAG, CAG, harness, evals, and guardrails fit in agent systems. Link: https://youtu.be/PMF81Nk4E6U . The original post and long article links are also here: https://x.com/Rel1vs337/status/2122720694786039967 , https://x.com/i/article/2116485158471671809 . Quoted from Livsun's article and tweet, as well as the Karpathy statement cited in the tweet.
发布时间: 2026-11-18 15:31