0.1% of Neurons Decide Whether a Large Model Says “No”

A video has been making the rounds on X.

The poster, lyra, captioned it with just one line: “lets find a new one shall we.” In the frame, the left side is a code editor, the right side is a 3D point cloud where each point represents the activation state of a neuron, and above there is a chat window. It looks like a debugger built for neural networks.

nisten, who retweeted the video, said it was the coolest ML software he had ever seen, and that it was free. He added: “I feel sorry for the well-funded but mediocre insiders who are about to just take someone else’s code without even giving credit.”

The real weight of this tweet comes from the paper lyra linked in a reply: Targeted Neuron Modulation via Contrastive Pair Search, from Nous Research. The paper proposes a method called contrastive neuron attribution (CNA) that localizes the “refusal” behavior of large models to an extremely sparse set of neurons.

After instruction fine-tuning, large models refuse harmful requests. But where the refusal mechanism actually lives was previously unclear. Mainstream steering methods, such as contrastive activation addition (CAA), intervene on the residual stream (the signal passed between layers) to change behavior. When the intervention strength gets large, the output degrades, producing repetition or gibberish.

CNA’s approach is more direct: feed the model two groups of prompts, one harmful and one harmless, record the activation value of each MLP (multilayer perceptron) neuron at the last token position, compute the difference, and take the top 0.1% by absolute value. The whole process only requires forward passes; it needs no gradients and no additional training.

The experimental results are telling. On JBB-Behaviors (the NeurIPS 2024 benchmark of 100 harmful prompts), after zeroing out the activations of the discovered 0.1% of neurons:

• Llama-3.1-8B-Instruct: refusal rate drops from 90% to 34%
• Qwen2.5-7B-Instruct: drops from 87% to 2%
• Llama-3.1-70B-Instruct: drops from 86% to 18%
• Qwen2.5-72B-Instruct: drops from 78% to 8%

For most models the refusal rate falls by more than half, and for some by more than 90%. At the same time, output quality basically does not drop. The paper’s comparison is: CNA maintains generation quality close to baseline (>0.97) at all steering strengths, while CAA degrades sharply once the strength exceeds 0.5, with several models collapsing directly into repetitive text. CNA also maintains baseline accuracy on MMLU, whereas CAA approaches zero at maximum intervention.

A more counterintuitive finding comes from comparing base models and instruct models. The same method can also find neurons with large activation differences in base models, and their locations are likewise concentrated in the last roughly 10% of layers. But zeroing out these neurons does not change refusal behavior; it only changes how the content is phrased. In other words, these “discrimination structures” already exist during pretraining, and instruction fine-tuning turns them into a switchable “refusal gate.” The paper’s authors call it: fine-tuning transforms function, not structure.

In the replies, some people asked which software this was; some marveled that it looked like a debugger for LLM networks; others said it was cyberpunk. Another user asked whether it could be used to modify model behavior, such as safety limits and refusals. lyra’s answer was: yes, followed by a link to the paper.

Incidentally, the demo video’s resolution is 992p, and someone took a screenshot specifically to ask: “what in the fuck is 992p.”

The paper says the code will be open-sourced at github.com/NousResearch/neural-steering. The full paper is at arXiv:2605.12290.