Reining in coding agents: Google's four-step checklist, and the comments argue over the order first
Google Cloud Tech uses four steps to explain how to build an AI harness, while the comments reorder where sandboxing, evaluation, memory, and authorization belong. Behind the debate over order is the seam problem agents can't avoid.
Google Cloud Tech posted a four-step checklist on how to build an AI harness for coding agents. The original post includes a 9-minute-27-second crash course, with the link at the end. The four steps are: context and memory, including sessions and state; tools and workflows, with mentions of MCP and ADK; sandboxing and prompt-injection defenses; and evaluation, split into code, LLM judge, and human.
The checklist is neat. The comments take it apart even more neatly.
One commenter thinks sandboxing should move to step one. The reasoning is blunt: injection defenses will eventually miss once, and after a miss, what matters is what permissions the agent holds. Another commenter puts evaluation first. When testing Rust, he hid the tests; if the model can see the tests, it may learn to pass the tests rather than solve the problem. That detail is more useful than "eval is important."
John Rood pulls memory out of the steps: the other three steps all write into it. A sandbox blocks a call, an evaluation catches a regression, a workflow sets a convention—when the session ends, all of that can be gone. So every harness run is like learning everything over again. He built memoryrouter. You can disagree with his product, but the observation is worth noting.
Other replies fill in the seams. HodorAI adds identity: the agent acts under its own identity, so permissions and auditing have something to attach to. José adds authorization for every tool call: the sandbox limits blast radius, but someone outside the model still has to decide whether this specific action can be performed; if there is no rule, deny it. Lachy asks more specifically: when an MCP tool returns attacker-controlled text, who handles it—the sandbox or the prompt defenses? 波若波罗蜜 suggests testing the seams specifically: a malicious tool response is first stored in memory, then retrieved in a later session. If this scenario isn't tested, the four steps are just four drawers.
Elroot Labs offers a hard metric for evaluation: look at the tool calls the agent emits; a polite refusal followed by a send_email still counts as a failure; and run each test case multiple times. The data he cites is NIST's AgentDojo work: repeating the same attack 25 times raised the average success rate from 57% to 80%.
Ekos Project asks about another layer: with sandboxes, tools, and evaluation all in place, how do you know the agent actually understands the codebase, business logic, and dependencies? If all technical evaluations pass but it breaks something it didn't know existed, what then?
Andrew says evaluation is the step everyone skips, while sandboxing is easy to sell. Put Vinay's hidden tests together with Elroot Labs' repeated runs, and you can explain why evaluation easily becomes decorative. Passing once doesn't mean stable, and seeing the tests doesn't mean it can solve the problem.
One judgment: the value of this checklist isn't in the order; it's that it reframes coding agents from a model-capability problem into an engineering problem. Once models can write code, what remains is the workbench, tool cabinet, isolation room, and acceptance sheet. The order can be argued over, but the seams between permissions, memory, evaluation, and auditing cannot be left empty. The neater the checklist, the easier it is for people to think that once it's installed, everything is settled. Original video: https://goo.gle/3XtB36o
发布时间: 2026-11-17 07:29