Wink Pings

Software Factory: After Agents Write Code, Humans Become the New Bottleneck

Ben Holmes argues that software engineering is becoming factory engineering. Agents have caused code output to skyrocket, but review has not sped up in tandem. Harvard data: review time +49%, PR rejection rate doubled. The real challenge is verification, not code generation.

Software engineering is turning into factory engineering. Ben Holmes wrote this judgment on X, and also published a long post explaining what a factory engineer is.

His core observation is: software development has always been divided into three steps (plan, implement, review). In the past, implementation took up most of an engineer's time, and planning and review could only happen in the gaps. Now agents have taken over the implementation step, freeing up engineers' time, but if you just make agents run faster, the work state turns into 20% meetings plus 80% busywork.

One commenter pointed out in a reply: every factory gets stuck at quality inspection. Agents open PRs (pull requests) 10x faster, but human review speed has not. Another commenter cited a Harvard study covering more than 700 companies: review time increased 49%, PRs sent back for revision almost doubled, and task completion did not improve. Another reply was more direct: use another agent to read conversation logs; scoring can only show what it said, not prove the changes actually run.

These doubts ground the concept of "factory engineering." Automation solved output, but not verification.

So what exactly does a factory engineer do? Ben Holmes's definition is: product engineers are responsible for the product itself; factory engineers are responsible for building the "agent system that produces the product." Specific problems to solve include: agents always like to add unnecessary test cases and comments—can we make them write better code; during on-call, repeatedly pasting error monitoring reports to agents—can this be automated; after the latest model is released, API bills spike—can we switch to a cheaper model without sacrificing code quality. The common point of these problems is system-level optimization, not one-off prompt tweaks.

A software factory is an automated loop. Ideally, you provide product requirements, the factory automatically builds and optimizes the product, and humans intervene only at necessary nodes. In practice, agents are not at that level yet. What factory engineers need to do is build a shared, centralized environment that both agents and humans can access, and that is configurable: runtime environment, permission level, workflow steps. A typical process goes from specification to PR, with human confirmation points set in between.

With a factory, you also need metrics. Ben Holmes proposed four categories: cost metrics—look at token consumption and compute cost, converted into cost/PR and cycle time; LLM-as-judge scoring—have another agent monitor past conversations and evaluate things that cannot be statically checked, such as architectural quality, redundant tests, and tool-call efficiency; self-improvement—expose metrics to agents so they can find problems and adjust themselves; benchmarks—change only one variable at a time to see which configuration has the biggest impact on cost and quality. Model selection is the biggest lever.

These four categories of metrics sound clear, but many people in the replies expressed skepticism. Someone asked: which metrics are actually related to agent output quality. Someone said: evaluation is too messy; evals are hard to do. This skepticism makes sense. If a factory only tracks output speed and ignores quality, it will ultimately only produce more half-finished work that needs rework.

Terminology iteration continues as well. A commenter joked: prompt engineering -> context engineering -> MCP engineering -> skill engineering -> loop engineering -> agentic engineering, and finally back to software engineer. There are more and more new terms, but the essence is still writing code and verifying code.

Infrastructure is also catching up. In the discussion, an open-source project called Gortex also appeared. It indexes code into a graph, provides queries to agents through MCP, and claims a 50x reduction in response tokens. It solves the same problem: let agents obtain only the context they need before acting, instead of reading the entire repository. This kind of tool is part of the foundation of factory engineering.

Gortex Web UI

Ben Holmes's team made a free course. Those who want to get hands-on can go to warp.dev/learn. But the course addresses how to build a factory, not the hardest verification problem.

The concept of factory engineering is not new. Its essence is bringing common sense from industrial engineering into software: automate low-risk work and focus human attention on high-risk judgments. The problem is that today's high-risk judgments (code review, quality assessment) are precisely the parts that have been least automated. Agents' code-writing speed has increased, but review is still manual. Making verification trustworthy may be more urgent than making agents write faster.

发布时间: 2026-11-20 19:59