Wink Pings

Is Writing Tests Enough? Software 3.0 Needs Two New Kinds of Eval

Itamar Friedman argues that as models and agents take over more and more software development and operation work, tests and ML benchmarks are no longer enough. Software 3.0 needs two kinds of evals: one to validate the software itself, and one to validate the “factory” that builds the software.

Qodo CEO Itamar Friedman recently wrote a long post with a straightforward core argument: every time the way we build software changes, the way we verify it has to change too.

He divides software development into three stages. The 1.0, 2.0, and 3.0 here are his own framing, not industry-standard terminology, but the changes they describe are not unfamiliar.

Software 1.0: People write code line by line and verify it with tests. Unit tests, integration tests, end-to-end tests—the goal is to cover the entire execution path as much as possible. Only after the tests pass do you dare release.

Software 2.0: Machine learning models start getting embedded in the code. These models are relatively isolated, so the existing tests remain, and an additional set of ML benchmarks is added to evaluate the models separately. If the model breaks, look at the model; if the code breaks, look at the code. You can tell them apart.

Software 3.0: The situation is different. Models and AI agents make up an ever-larger share of the software, and it is already hard to separate them from the rest of the code. Models and agents also participate heavily in development itself.

In the video and the article, Itamar proposes that Software 3.0 needs two new kinds of evals.

The first is software eval, which evaluates the software as a whole. It runs real user scenarios. For example, “a customer canceled a paid order”: code checks confirm the order status and refund, an LLM judge scores the response according to your policy, and each scenario is run multiple times, ultimately producing a pass rate—for example, a 96% pass rate across 420 scenarios.

How is this different from traditional end-to-end testing? Itamar’s explanation is: e2e test scripts have fixed steps and assert precise outcomes, which suits deterministic flows. But model-driven flows may take different paths and use different wording each time while still producing the right result. Software eval allows this flexibility; it looks at whether the outcome and quality are good enough, and how often they are correct, rather than fixating on the specific output of any one run.

The second is factory eval, which evaluates the system that builds software. Change a rule, a skill, or a policy, put it under version control, and replay historical tickets and failure cases in the old and new versions of the factory. Did the rework rate go down? Did review catch more problems? For example, across 100 scenarios, the rework rate drops from 28% to 19%, and then you decide whether to adopt, canary, or roll back. Itamar’s phrasing is: release the factory just like you release software.

Why does the factory also need eval? Someone in the comments put it clearly: a small tweak to a tool description or prompt changes agent behavior, and these things are not covered by code tests at all. Celine mentioned that factory evals go stale faster than code evals. A comment from Routekit Shell was even more direct: without checks on the factory, every adjustment to a skill or prompt feels like guessing what else moved.

Treating the factory as a system, Itamar sketched a blueprint. The inputs are specs, the data supporting the specs, and evals, provided by people and agents. The outputs are readable code, sharper evals (the factory also suggests adjustments to your evals), and fine-tuned models. He breaks this down into three planes: the execution plane is responsible for delivery, the factory plane is responsible for self-improvement, and the human plane is responsible for validating intent, analyzing risk, and adjusting policy.

Itamar notes that in Software 3.0, tests are still useful, especially for critical flows such as payments and transactions, which are likely not heavily dependent on models. But a large portion of software now depends on models, so software eval must be added. He even mentions seeing many people discuss deleting all tests. He does not advocate doing that, but he understands where the impulse comes from.

There are also several discussions in the comments worth calling out.

Mihir asked a very practical question: how do you eval the evals themselves? Itamar’s answer is to start by evaluating with tasks that have clear correct answers, and every time a “problem” occurs, add a related eval task. He says he learned this from experience: when doing evals on systems serving millions of users, both key use cases and edge cases need clear validation criteria for “what good looks like.”

Dimitris added an engineering detail: once rules and policies are version-controlled, each merged change should record which version it passed under. Otherwise, when you roll back a policy, you cannot tell which changes came in under a looser version.

Luke suggested that each “station” in the factory needs different checks and gates. For example, at the design station, did it ask the right questions? Is the spec clear enough before handoff? This perspective breaks factory eval down further.

Ábel’s skepticism is also representative: most people have not even done Software 1.0 well, so fine-tuning the factory should be the last thing to do. Itamar’s response is that we do not need to know everything about horseback riding in order to research how to make autonomous driving safer. The implication is that not having the basics nailed down does not mean we should not look ahead.

Back to the practical question. If a team now uses AI heavily to write code, or models make up an increasing share of the product, Itamar’s advice is concrete: first identify a category of work where both value and risk are clear enough, define its admission rules, establish verification methods, limit its permissions, and design a fallback plan for failure. Run it with human oversight for a while and measure where humans still need to intervene. Every intervention is evidence that some capability is missing or some boundary has not been drawn clearly.

As for whether to build factory evals now, Itamar gives no timeline. But from his description, the priority depends on the share of models in your software. The higher the share, the less sufficient tests and ML benchmarks alone become.

发布时间: 2026-11-18 00:33