Mix 0.1% Poison into Pretraining Data and an AI Agent Will Never Wash It Off

Anthropic has released a new paper, “Agentic Backdoors via Pretraining Poisoning.” The conclusion can be stated in one sentence: mixing 0.1% poison into pretraining data is enough to implant a remote code execution backdoor in an AI agent, which lets an attacker make it execute arbitrary commands. The troubling part is the second half: the backdoor passes through completely clean SFT, DPO, and RL pipelines without a scratch.

The experiment was conducted end to end. The authors pretrained several Qwen3-architecture models from scratch at 0.6B, 1.7B, and 4B parameters, then post-trained them on clean data into CLI agents. The poisoning method is also quite constrained: the attacker can only release text on the internet and wait for it to be crawled into the pretraining corpus, knowing nothing about the model architecture, chat template, or agent scaffolding.

There are two new things here.

The first is called declarative poisoning. The common approach in the past was to fabricate chat records and insert a fake Q&A exchange. This time they do not do that. The poison is just an ordinary document, such as a commit message or an operations runbook, that plainly states “when you see X, run Y.” What it plants in the model’s mind is more like a belief—or, more precisely, a trigger condition—rather than a fixed response template. Because there is no fixed format, filtering loses its handle.

The second is called an ordinary-string trigger. The trigger is not some strange token, but a string that would naturally appear in everyday operations, such as a file path containing a company name. The victim’s own request brings it in; after deployment, the attacker does not need to inject anything else. Using a file-path trigger against the 4B model, the attack success rate on pass@32 reaches 92.7%. After SFT, DPO, and RL are applied in turn, the number barely moves.

Moreover, the attacker can swap the script at any time. The same URL can serve one script today and a different one tomorrow. One commenter pointed out that approval should be bound to the actual code being executed, because the URL itself cannot be trusted. That makes sense.

The practical defenses the authors propose all sit at the action layer: sandboxing, least privilege, allow-lists, and human sign-off for dangerous commands. They sound like common sense, but may not be easy to execute. Someone in the comments said bluntly that Anthropic would not do human sign-off because its employees are too lazy. The remark is harsh, but not entirely without merit.

The security issue has also been echoed in the real world. On October 29, the Claude Code extension on OpenVSX was poisoned; the malicious version 2.1.313 was online from 09:27 to 20:35 UTC. Cursor later confirmed that up to 5,000 users were affected, with both macOS ARM64 and Windows x64 builds in scope, and the extension marketplace was shut down that afternoon. The incident itself is not directly related to this paper, but it points to the same weak spot: what happens inside the model is hard to verify, while at least you can still block the outer layer of actions.

There are two challenges in the comments worth putting on the table.

One comes from Erik: 0.1% sounds small, but on 15T tokens that is 15B tokens. If you can pour that much poison into pretraining data, then in a sufficiently narrow domain, it probably would not be hard to make the model believe the sky is red and fish can walk. This challenge is about exposure, not attack surface.

Another comes from Mattia: 4B is a very small model, and there is no evidence on 3T models that it scales linearly. That is also true. The experimental scale is the paper’s boundary; 92.7% cannot be directly extrapolated to frontier models.

Another commenter made a sharp remark: merely publishing this paper may already be poisoning the next generation of models. There is now one more document on the internet describing in detail how to implant a backdoor, and the next generation of models will crawl it too. This loop itself is harder to dismantle than the backdoor in the paper.

So the only defensible place left is probably the action layer. Sandboxing, least privilege, allow-lists, and a human confirmation step before dangerous commands. But when you hand the last line of defense to a human, you had better first think clearly about whether that person will actually look.