Agents Keep Falling Apart—Don't Switch Models Yet

A tweet was spreading quickly through the timeline.

The claim was catchy: Andrej Karpathy spent 8 years at OpenAI and Tesla, and last week he compressed what he learned into a free 2-hour lecture, with the path Agents → Harness → Loops → Graphs. The caption added that some bootcamps charge $15,000 and teach less than this.

The comments were less polite. Some asked where the link was, some said it was spam or selling a course, and others directly corrected it: don't say "last week"—this material has been around for two years.

The "last week" in the tweet really doesn't hold up. But the long article it led to is worth reading. The author is Lunar, and the title translates literally as "Harness Engineering: A Complete Guide to Building AI Agents That Don't Fall Apart."

The article opens by pointing to an often-overlooked layer: most people fixing agents are fixing the wrong layer.

When an agent fails, they first change the prompt. When it fails again, they add more instructions. Then they switch models, add tools, expand the context window, and hope the next round will be different.

Lunar's judgment is that many failures are not reasoning failures, but environment failures. The agent doesn't know which files matter; it uses the right tool but in the wrong place; decisions made in the previous session are lost; it says it's done without running checks; after a partial failure it repeats the same action; permissions are so loose that it can touch things it shouldn't. The model may not be the problem; the system around the model is incomplete.

This system is the harness. Its definition isn't mystical: it's the environment that turns the model's intelligence into reliable work.

A prompt changes one attempt; a harness changes every attempt.

Here are some of the harder points from the article.

A model is not an agent. A model can reason, generate, compare, and choose. But an agent also has to deal with the real environment: understand the task, find context, choose tools, preserve state, enforce permissions, inspect results, recover from failures, and prove the work is done. A strong model wrapped in a weak harness is still a weak agent. A single-turn answer may look great, but stretch it to a multi-step task and it starts to drift.

Write the task contract first. Most tasks start as a vague intent, like "optimize the onboarding flow." That sentence is enough for chat, but not enough for autonomous execution. The harness should first translate the request into a contract that answers five questions: what result must be produced, what is in scope, what must not be touched, what evidence counts as completion, and which actions require human approval. Without a contract, the agent optimizes for "looking like it's working"; with a contract, it can optimize for "verified completion."

Give it a map, don't stuff in a manual. Dumping the entire repository, all documentation, and all conversation history into the context is not context engineering; it's a context flood. A better approach is to give a small map first and fetch details when the task needs them. Context should expand because the task requires it, not because the information exists.

Tools should go through a gateway, not pile up. Giving an agent twenty tools doesn't make it more capable; it gives it twenty ways to make mistakes. A tool gateway manages: hiding irrelevant tools, validating parameters, restricting paths and domains, adding timeouts, making retries idempotent, requiring confirmation for dangerous actions, and returning evidence instead of just a "success." The model can propose an action; the gateway decides whether that action is qualified to execute.

Completion must rely on evidence. When the agent says "done," it's just another model output. Completion must be judged by observable changes in the environment. Run the cheapest deterministic checks first: compiler, schema, checksum, query, tests. The model handles ambiguity; code handles the pipeline. The model can propose that the task is complete; only the environment can prove it.

Verification should attack the result, not give a second opinion. The worker and the evaluator shouldn't share the same goal. The worker wants to build the strongest solution; the evaluator wants to find reasons it should be rejected. Asking the same agent in the same context to "check it again" will most likely preserve the assumptions that created the error. The verification stage needs explicit rejection criteria, access to artifacts, access to the acceptance contract, independent tools or a new context when necessary, and the authority to "reject without being responsible for fixing."

Recovery should be based on failure type. The most common approach is: it failed, try again. That's not recovery; it's repetition. The harness must first classify the failure, then choose the next action. A retry must change at least one relevant condition; otherwise you're paying to reproduce the same failure. Every loop needs a budget: maximum attempts, maximum time, maximum spend, maximum blast radius, escalation conditions.

Harnesses rot too. Models get stronger, tools get better, tasks change. A patch made yesterday for an old model may be blocking a new model from using a better strategy today. Every router, evaluator, memory layer, and retry rule should be asked: what failure is this preventing? Is that failure still common now? How much latency and complexity does it add? What happens if we remove it? The best harness isn't the biggest one; it's the smallest system that reliably closes the gap between intent and evidence.

The article ends with a shift: the first generation of AI products was built around prompts; the next generation will be built around environments. The question changes from "how do we make the model answer better" to "how do we build a system where good actions are easy to take, dangerous actions are controlled, failures are visible, and completion is provable."

Models provide intelligence; harnesses provide structure.

If the agent you're working with is still surviving on extra adjectives, don't rush to switch models. Take a look at the system around it.

Original link: https://x.com/i/article/2096536117130481673