Software engineering is becoming factory engineering, but the bottleneck is still QA
Ben Holmes says software development has evolved from writing code to writing prompts to designing an entire agent system. He calls this 'factory engineering.' How do you build a software factory, what metrics do you use to measure it, and why is the comment section arguing that 'QA can't keep up'? This article lays out the concepts, methods, and doubts together.
Ben Holmes published a long post on X with a blunt title: software engineering is becoming factory engineering. When Zach Lloyd reposted it, he added that it is a great introduction for anyone who has just heard of 'factory engineering.'
Ben Holmes is from Warp. His core thesis is that software development has undergone three shifts: from writing code, to writing prompts, and then to designing an agent system that runs on top of prompts. This is no longer a simple coding problem but a system-design problem, and it has brought new energy to the entire industry.
He breaks software development into three stages: planning, implementation, and review. Before agents appeared, implementation was the main body of deep work, and only the remaining time was left for architecture discussions and code review. Now agents handle most of implementation, and the balance has flipped. If you only use the time saved to go faster, you may feel that work has become 20% meetings and 80% miscellaneous tasks.
But behind this tedium, new engineering questions arise. During planning, at what points should the agent pull humans in? During implementation, how do you balance code quality and cost, and which secrets, MCP tools, and skills can the agent access? During review, how can the agent first verify its own work and attach a risk assessment to the PR, instead of dumping all 10x-speed PRs on humans?
Ben Holmes therefore defines two new roles. The product engineer decides what the product needs and how it should be architected. The factory engineer is responsible for building the software factory, an automated system built around agents. It does not aim for full automation; instead, it automates low-risk steps and preserves human intervention only at key points. In the example workflow he gives in the article, the spec and the pull request are the two main human checkpoints.
What does a software factory look like
Simply put, it is a shared environment and process for humans and agents, usually hosted in the cloud or on unified compute nodes rather than on each person's local machine. You can configure what environment the agent runs in, what permissions it has, and what steps it follows to complete work. The larger the product and the more agents deployed, the more obvious the value of factory engineering.
How do you measure the factory
Ben Holmes offers four directions.
- Cost. Watch token usage and compute costs, but more importantly translate them into real value, such as the cost per PR and the cycle time from the initial prompt to release.
- Scoring by another agent. Static checks cannot cover metrics such as architectural quality, the proportion of redundant tests, and tool-call efficiency. A 'judge' agent goes back through the session logs afterward and assigns scores. But some commenters pointed out that the judge reads the conversation logs, which can only prove whether the agent talks well; in the end, you still have to run the changes to see the results.
- Self-improvement. Expose the scoring results to the agent. Suppose a 'redundant tests' scorer performs poorly in 60% of sessions; a self-improvement agent would notice this trend and propose adjustments to skills or code architecture.
- Benchmarking. Treat configuration changes as experiments, changing only one variable at a time, such as swapping models, adding or removing stages, or adjusting the number of sub-agents, and observe how the metrics change.
The comment section started arguing
After the article was published, many replies raised doubts.
A commenter named Juan Sanchez said that every factory he has seen gets stuck at QA: agents submit PRs ten times faster, but human review does not get ten times faster. Even if low-risk PRs can be auto-merged, how do you improve the experience when human sign-off is required?
Another commenter named Paul asked more directly: where is the evidence? There are no success rates, probabilities, or data proving that the current model is optimal.
Another reply cited a study allegedly from Harvard covering more than 700 companies: review time increased by 49%, the number of PRs sent back for changes nearly doubled, and the number of completed tasks did not change.
These doubts do not necessarily overturn the 'software factory' framework, but they at least show that it currently looks more like one company's view based on Warp's practice rather than a mature paradigm backed by ample evidence.
If you want to verify it yourself, Warp, where Ben Holmes works, offers a free public course, warp.dev/learn, that teaches how to build a software factory from scratch. If you want to test these doubts, you can start with this course.
发布时间: 2026-11-20 05:46