GPT-6 Astra Directly Edits Weights: A 0.016% Change Lifts a 0.8B Model's Math Score from 7.9% to 48.1%

Rikiya Takehi and colleagues at MIT CSAIL released a set of experimental results on November 2: letting GPT-6 Astra directly read and write neural network weights. No gradients, no training loop, and no asking the model to write its own training script. The authors call this Weight-Space Programming.

What are weights? For general readers, think of them as all the adjustable numbers in a model. Traditionally these numbers are updated by an optimizer according to gradients. Weight-Space Programming takes a different direction: it hands the weights to a general-purpose LLM as text or tensors, and lets it read the structure, find faults, and directly write new values. Astra can use Python to read weights and do matrix multiplication analysis; in some experiments it can also do a limited number of forward passes, i.e., run the network forward once. It cannot train, cannot fit data, and cannot search trees.

Experiment 8 is the most striking. The target is the mathematical ability of Qwen3.5-0.8B, where 0.8B means about 800 million parameters. This model scores only 7.9% on MATH-500, and Qwen's model card warns that it has a thinking loop problem. With a budget of 50 forward passes, Astra located a certain direction written by the MLPs of layers 15 and 19, subtracted this direction from each layer's 64 output columns, and changed a total of 131,072 weights, or 0.016% of the 0.8B weights. No gradients, no training. As a result, MATH-500 rose from 7.9% to 48.1%, GSM8K from 13.8% to 67.1%, and IFEval from 21.0% to 62.3%. The proportion of MATH-500 hitting the 32,768-token limit fell from 93.0% to 33.1%. With 10 and 25 forward passes the effect was much worse, indicating that the budget and diagnostic process matter.

Saying training is useless would be overkill. The authors themselves provide a counterexample: Gemma 3 270M rarely loops, and when Astra read the failure case it saw only ordinary arithmetic and algebra errors, with no single component to turn off. Its edits moved MATH-500 by at most about one point, and Astra itself reported that this gain was not established. Being able to remove a fault that blocks a capability and adding a capability from scratch are two different things. This boundary is more worth watching than the numbers.

The weight-reading part is also interesting. Experiment 1 used PPO to train four game networks of the same size, with identical inputs, outputs, and layer widths; Astra was given only the text of 17,253 weights, without being told the candidate games, and in all four cases it reported the correct game name. Experiment 2 judged from DDPM weights whether the generated shape was a circle, square, or triangle; it drew the shape as an outline, which looks more like reading the output-layer structure than relying on a forward pass. Experiment 3 removed the first and last layers of a small language model and gave Astra only four transformer blocks; it could still determine whether the training corpus was Python, romanized Japanese, or English, and identify the copying head. The method it used later corresponded to circuit analysis by Elhage et al., but the prompt did not mention that work.

The weight-writing part is more like a stress test. Experiment 4 gave a PPO network stuck on the Kinetix ramp task; Astra looked at the weights, the scene, and a trajectory, and in one edit changed 3.5% of the weights to make the car first reverse and then charge up the ramp, succeeding in all 1,024 episodes, whereas the original network failed in all 256 episodes. Experiment 5 wrote MinAtar networks from scratch, with no data, loss, gradients, or tools, and without letting the network try the game first. On Breakout, Astra's one-shot generated network scored 517.4, versus 15.8 for PPO at 100M steps and 25.3 for PPO at 500M steps; on Seaquest it scored 53.1, versus 88.9 for PPO at 500M steps; on Asterix it scored 53.2, versus 55.2 for PPO at 500M steps. Experiment 6 wrote a 9x9 Go network with 1.48M weights; with 0 forward passes it could win more than half its games against two small PPO networks, and after 50 forward passes it could beat a same-sized PPO network trained for 20M games.

There were failures too. Experiment 7 tried to make Astra judge, using only Gemma 3 270M weights, which model had been fine-tuned to cater to users. All four attempts failed; it could find weight differences but could not map them to behavioral differences. When relaxed to allow it to write text and look only at the two models' attention patterns, it chose correctly for all three model pairs. This detail shows that pure weight reading is not yet universal, and that tools and interventions change the capability boundary.

At the end of the blog, the authors list possible directions: auditing backdoors or hidden capabilities without running evaluations, reducing optimization costs, adding capabilities or forgetting with zero data, model merging, and recursive self-improvement. That sounds big. For now, the more accurate judgment is: LLMs are beginning to make local, interpretable edits to weights, especially when the fault is clear, the network is not large, and the budget is controlled. It is still far from replacing large-scale training. What is really worth tracking is whether this approach can become a reproducible engineering workflow. A few impressive demos are not enough.

Links: blog https://rikiyat.github.io/blog/weight-space-programming/ , full logs https://github.com/RikiyaT/lm_understand-logs , weights for Experiment 8 https://huggingface.co/RikiyaT/Qwen3.5-0.8B-weight-space-programming , demo https://huggingface.co/spaces/akhaliq/weight-space-programming-demo .