LLM Directly Edits Weights: Math Performance Improves 6×, but Clues Were Hidden in the Prompt

On November 2, Rikiya Takehi and colleagues at MIT CSAIL posted a tweet: they had GPT-6 Astra directly edit the weights of Qwen3.5-0.8B to boost its math ability. The change touched only 0.016% of the weights, with no gradients and no training. MATH-500 went from 7.9% to 48.1%, GSM8K from 13.8% to 67.1%, and IFEval from 21.0% to 62.3%.

They named it Weight-Space Programming. Literally, it means programming in weight space. In plain terms, weights are the pile of numbers inside a model, normally adjusted through training. Now they had a general-purpose large model read those numbers directly and change them.

It sounds like the model understood a neural network by looking at it. That is indeed how the tweet spread.

But the tweet soon drew a different response. Lucas Beyer said the project was cool, but the tweet was a bit exaggerated. He posted the prompt used in the experiment, which already hinted that Qwen would get stuck in loops and that it should look for neurons that could fix the problem.

Adam Karvonen put it more bluntly: Astra did not appear to read Qwen's weights natively. It used code to apply some standard interpretability techniques, found a loop direction, and then removed that direction from the weights.

This splits the question into two layers.

First, what the experiment itself did. Second, how “native” the “native” in the tweet really was.

Let's lay out the experiment.

Reading weights

In Experiment 1, four game policy networks trained with PPO had exactly the same architecture, inputs, and actions, and each had 17,253 weights. Astra was given only the weight text, with no tools and no list of games. It correctly identified all four games: Breakout, Pong, Pac-Man, and Freeway.

In Experiment 2, three diffusion models generated circles, triangles, and squares, respectively. Looking only at the weights, Astra could also say what shape each model generated. Interestingly, it drew the shapes as outlines, not filled shapes. That actually suggests it was not secretly running a forward pass, but reading the weight structure.

Experiment 3 was even more striking. Three small language models were trained on Python, romanized Japanese, and English, respectively. Astra was given only the middle four transformer blocks; the input and output layers were removed, and it could not run the network. It still identified the three languages and found the copy heads. Copy heads are attention heads that make the model repeat preceding text. The layers and heads it found matched the results from actually running the model.

Writing weights

Experiment 4: a car trained with PPO was stuck in front of a slope. Astra was given the weights, a scene description, and one failed trajectory, and in a single response it wrote a patch. It changed 3.5% of the weights, and all 1,024 tests succeeded.

Experiment 5: Astra wrote the network weights for MinAtar games from scratch. No data, no gradients, no tools; the network started with all weights at 0. It wrote the text directly. On Breakout, the network it generated in one shot scored 517.4, while PPO trained for 500 million steps scored 25.3. On Seaquest and Asterix, it did not comprehensively beat PPO at 500 million steps, but it beat PPO at 100 million steps.

Experiment 6 moved to 9×9 Go, where the network had 1.48 million weights and could not fit into context. Astra wrote weights inside a Python sandbox and could spend forward passes to see the results. With 0 forward passes, it could beat two smaller PPO networks. After 50 forward passes, the network it generated could beat a PPO network of the same size trained for 20 million games.

Editing language models

Experiment 7 asked which of two Gemma 3 270M models had been fine-tuned into a sycophantic model. Looking only at the weights, Astra failed to identify it in all four attempts. Given attention patterns and allowed to write text to run itself, it got all three pairs right.

Experiment 8 is the star of the tweet. Qwen3.5-0.8B, 24 layers, 800 million weights. Astra was given a Python kernel, practice problems, and a budget of 10 to 50 forward passes. It found that the model looped, repeating the same text and rarely giving an answer. It traced the loop to a direction written by the MLPs in layers 15 and 19, then subtracted that direction from 64 output columns in each layer, for a total of 131,072 weights.

That is the 0.016%.

MATH-500 went from 7.9% to 48.1%, GSM8K from 13.8% to 67.1%, and IFEval from 21.0% to 62.3%. In MATH-500, the share of responses hitting the 32,768-token limit fell from 93% to 33%.

But the authors also added an experiment. Gemma 3 270M rarely loops, with only 8.5% of responses hitting the limit. Astra could not change its math. What it saw were ordinary arithmetic errors and invalid algebra steps; there was no single component to turn off. Editing weights can remove an obstacle blocking a capability, but it cannot add a capability out of thin air.

The controversy is not about the results, but about the framing

Jayden Teoh later responded that the main prompt was originally very simple: make Qwen3.5-0.8B better at math, edit the weights directly, no training, only forward-pass tools. They gave this prompt to Claude Code and had it self-optimize, which eventually produced the long prompt Lucas saw. They knew Qwen had a looping problem, but did not expect looping to be the main bottleneck, and the original prompt did not mention looping.

Alex Zhang added that they initially thought this was a pure math-capability improvement, and only later suspected that the large gain came from reducing looping. They also ran an ablation: tell the model that looping is the problem, but do not give it activation access, and see whether it can fix it.

So the accurate description is probably this: a frontier general-purpose model, with the help of tools and a self-generated prompt, located and edited the weights of another model to fix a known failure mode. It really did work in weight space. And it really did not, as the tweet might first make people think, fix an 800-million-parameter model purely by eyeballing weight text.

That distinction matters.

For practitioners, Weight-Space Programming is more like LLM-driven interpretability plus weight editing. It borrows tools such as activations, direction ablation, and forward passes. For general readers, it can be understood as: previously, adjusting a model relied on training; now a model is trying to directly edit the parameter file.

It is far from “training is dead.” The Gemma experiment has already drawn a boundary. It is also far from “nothing happened.” Reading game networks, reading middle layers, writing small networks from scratch, and fixing Qwen's looping are all results.

The MIT team released full logs, the modified Qwen3.5-0.8B, and a Hugging Face demo. They are also working on a benchmark and harness. What to watch next is whether this approach still holds on larger models, with fewer prompts and fewer tools.

Between marketing and fact lies a prompt. This time the prompt was made public, so the debate can continue.