Neuron Lens Lays Model Circuits Bare, and Activation Flow Is Often Readable

Neuron Lens has launched a new feature: circuits. Developers a9lim and treetown traced how activations flow between different features, and concluded that these paths are often quite legible.

The tool could previously already inspect the internal responses of two open-source language models, Qwen3-8B and Gemma 4 12B, by neuron and attention head: what words and concepts each unit is most sensitive to, which examples make it fire hardest, and what the model says after steering it. For Gemma 4, you can also see the images and sounds it responds to. Each unit has an address, such as /qwen3-8b/L8/n344 or /gemma-4-12b/L35/h14. The Explorer page shows 442,368 neurons, 1,152 attention heads, and 36 layers. Labels are generated by Claude Haiku 5.5; readings come from weight lenses, top firings, and steering checks.

The new thing now is circuits. According to a9lim, how activations flow between features has been traced, and it is often legible. Tweet author roon believes this kind of work will lead to better lens techniques than blind reconstruction, and, if all goes well, will be more effective than CoT monitoring. CoT monitoring looks at the reasoning chain the model writes out, with a layer in between. The circuit view tries to directly see how internal activations travel.

A comment from user AJMUL is blunt: someone is finally mapping the weight mud instead of guessing. This points to the reality of interpretability tools. The inside of a model has long been like a mass of weights; researchers can measure behavior and see outputs, but it is hard to say which layer a concept is in, which unit pushes it, and where it flows next. Neuron Lens puts each unit's responses, strongest trigger examples, and steering effects in one interface, at least giving troubleshooting an entry point.

The website is a static single-page app built with Vite, Svelte 5, and TypeScript, with no server-side code. The data consists of several GB of static JSON; Gemma 4 also includes image and sound bundles, hosted on Cloudflare R2. The source code is open-sourced on GitHub under AGPL-3.0, and running it locally requires Node 24 or newer. The design uses Ledger, and the font is Wix Madefor. The authors are credited as a9lim, treetown, and Claude.

What's interesting is how divided the community reaction is. Some ask what is needed to reproduce it locally, mentioning a lot of exploration and computation, as well as validation. Some first reaction is that this could become a better control tool. Danmar's reply spells out the tension clearly: sharing more increases legibility for everyone, including AI itself; but every share can also become a means of enforcement. Monkey sees another side: this kind of verification may be less intrusive into user prompts, and inference providers could be required to have certain interpretability rather than watching what each user inputs. AvantGarde's worry is more direct: could this be used to delete entire concepts, censoring the model at the root, intercepting and rewriting outputs whenever it encounters deities or gods.

These worries are not unfounded. The better interpretability tools get at locating concepts, the easier they can become switches for editing concepts. Neuron Lens currently shows responses and circuits, not an automatic censorship system. But the boundary of a tool's capabilities and the boundary of its uses are often decided by the people who use it. For ordinary readers, the site has search terms such as owl, Bible, money, 圣经, and L20h17 that can be clicked and explored. For developers, GitHub has data-building scripts and deployment scripts. If you want to verify whether it is useful, the most direct way is to find a neuron address you are familiar with and see whether its trigger examples and steering results line up.

Original link: https://x.com/tszzl/status/2120757053295862195

Tool link: https://neuron-lens.com

GitHub: https://github.com/stdlogits/neuron-lens.com