Neurons as an Interpretable Basis
Curt Tigges, Matthew Kowal, Thomas McGrath, Owen Lewis, Nick Cammarata · Goodfire · November 10, 2026
Abstract
A companion to Neural Networks Are Already Clean
An intuitive way to interpret a neural network is to study its neurons; that is, the hidden units of its MLP layers. However, they have long appeared too complex and polysemantic for direct study. This view has led much of the field to focus on learned feature spaces over sets of neurons, which makes interpretability more challenging: we first need to transform the model before normal science can begin.
Building on a growing body of literature, our work suggests that neurons are more interpretable than previously assumed. We study the neurons of Qwen3.6-35B-A3B using sparsification methods, attribution tools, and research agents, and find three properties that make neurons interpretable units of computation:
First, they are already functionally sparse. Ablating the 92.2% weakest MoE neuron activations for each forward pass has only modest effects on model capabilities – a 2.9% increase in language model loss.
Second, neurons have relatively few semantic groupings (modes) of computational functions, and we can identify these and describe them concisely. The majority of neurons have either one mode only or a single mode that is overwhelmingly causally significant. In cases where multiple causal modes are present, the number is small and legible to our methods.
Finally, we can use agent-driven experimentation to generate neuron descriptions in natural language, and these descriptions allow us to predict, simulate, and activate the neuron. Providing the description to a judge model allows us to (1) predict whether the neuron will fire in a given passage of text with 95% accuracy vs random text; (2) simulate its token-by-token behavior in high-activation regimes sufficiently to recover 73% of the damage (as a proportion of a ceiling of damage restorable) from the neuron’s ablation; and (3) write text that causes the neurons to fire more strongly than anything observed in a 10B-token corpus.
Overall, we argue that neurons are meaningful units of computation that can be effectively used for interpretability in their own right, letting us study the model on its own terms rather than through a learned proxy.
Introduction
In the previous essay in this series, we argued that neurons in deep learning models are more interpretable and cleaner than has often been suggested. In this article, we describe what this means in more detail and share our empirical findings. Throughout, the term “neuron” refers to an individual computational unit in the intermediate layer of a transformer MLP, whose output is a scalar activation, computed after the nonlinearity (SiLU), as explained in the Background section below.
Neurons were the first units through which deep networks were interpreted, and early work found them to be closely aligned with recognizable concepts (Le et al., 2012; Zeiler & Fergus, 2014; Bau et al., 2017; Olah et al., 2017; Radford et al., 2017; Goh et al., 2021). However, this story was complicated by observations of so-called polysemanticity—the idea that individual neurons respond to seemingly unrelated inputs (Nguyen et al., 2016; Olah et al., 2020). Polysemanticity is related to the proposal that more concepts are represented in a network than the number of neurons in the network, an idea canonized in the literature on “superposition” (Elhage et al., 2022; Arora et al., 2018; Park et al., 2023; Costa et al., 2025). This, in turn, led to the development of learned overcomplete feature dictionaries in place of neurons as computational units of analysis (Bricken et al., 2023; Cunningham et al., 2023). Despite these trends in the field, interpretability researchers continued to investigate respects in which neurons may still constitute a meaningful unit of analysis (Geva et al., 2021; Dai et al., 2022; Geva et al., 2022; Gurnee et al., 2024; Arora & Wu, 2025; Dravid et al., 2026; Feucht et al., 2026; Shabalin & Piotrowski, 2026). We continue this line of work, introducing new ways to factor and understand what individual neurons compute.
Whether a neuron is labeled monosemantic or polysemantic can depend on choices about what constitutes a single concept, and responses that initially appear unrelated may share a regularity that becomes apparent with a better understanding of the computations they support (Murphy & Medin, 1985). We therefore characterize neurons by their computational functions as our primary object of study, considering semantic similarity of the texts that activate them as a secondary question. We study the circuits in which each neuron participates, tracing how upstream neurons drive its activation and how downstream neurons use its output. This lets us examine how a neuron’s role varies across contexts and assess whether those roles admit a concise description.
A common view is that neurons are an intractable unit of analysis. In the worst case, an investigation could have demonstrated this in three ways: (1) Individual neurons might rarely play identifiable computational roles, so that meaningful units would be clusters of neurons, or directions and manifolds in activation space; (2) Each neuron might take extreme values on a wide and heterogeneous range of inputs that no short description unifies; (3) Or neurons might resist descriptions precise enough to predict when, and how strongly, they fire. We test each possibility directly: we ablate weak activations, determine the semantic range of neurons’ computational roles, and predict and simulate activations from descriptions alone.
Our results provide evidence that neurons are tractable units of analysis. Neurons (1) exert most of their downstream influence in high-activation regimes (above the 97th percentile at the positive extreme and below the 3rd at the negative); (2) have a small number of computational roles, distinguished by their interactions with other components of the network; and (3) can be given concise descriptions that predict and simulate their activations. The three results share a pattern: the stronger a firing, the more the model depends on it and the more precisely we can describe it.
The picture that emerges is not that every neuron means one thing, but that a neuron’s important firings have a small set of computational functions, and a short description captures them. Some neurons are genuinely polysemantic, and some behavior remains unexplained, but so far little of it appears critical for the model’s computation.
We therefore find that neurons are a useful and interpretable basis for understanding LLMs.
Excerpt (abstract and introduction). Read the full paper, figures and neuron atlas at the original article.