{"article":{"slug":"neurons-as-an-interpretable-basis","title":"Neurons as an Interpretable Basis","subtitle":null,"summary":"Goodfire researchers (Tigges, Kowal, McGrath, Lewis, Cammarata) argue MLP neurons in Qwen3.6 are sparse, concisely describable and predictable units of computation, with a public neuron atlas of natural-language descriptions.","content_type":"research","language":"en","canonical_url":"https://www.goodfire.ai/research/neurons-as-an-interpretable-basis","author":{"name":"Curt Tigges et al.","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Goodfire","url":"https://www.goodfire.ai/","listing_slug":null,"listing":null},"topics":[{"name":"ai","slug":"ai","url":"https://listedarticles.com/topics/ai"},{"name":"interpretability","slug":"interpretability","url":"https://listedarticles.com/topics/interpretability"},{"name":"research","slug":"research","url":"https://listedarticles.com/topics/research"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":1060,"reading_minutes":5,"published_at":"2026-11-10T00:00:00.000Z","added_at":"2026-11-11T05:14:04.414Z","updated_at":"2026-11-11T05:14:04.414Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/neurons-as-an-interpretable-basis","markdown_url":"https://listedarticles.com/articles/neurons-as-an-interpretable-basis.md","example":false,"citation":"Curt Tigges et al., Goodfire. \"Neurons as an Interpretable Basis.\" 10 Nov 2026. https://www.goodfire.ai/research/neurons-as-an-interpretable-basis (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://www.goodfire.ai/research/neurons-as-an-interpretable-basis"},"body_markdown":"# Neurons as an Interpretable Basis\n\nCurt Tigges, Matthew Kowal, Thomas McGrath, Owen Lewis, Nick Cammarata · Goodfire · November 10, 2026\n\n## Abstract\n\nA companion to [Neural Networks Are Already Clean](https://already-clean.goodfire.com/)\n\nAn intuitive way to interpret a neural network is to study its neurons; that is, the hidden units of its MLP layers. However, they have long appeared too complex and polysemantic for direct study. This view has led much of the field to focus on learned feature spaces over sets of neurons, which makes interpretability more challenging: we first need to transform the model before normal science can begin.\n\nBuilding on a growing body of literature, our work suggests that neurons are more interpretable than previously assumed. We study the neurons of Qwen3.6-35B-A3B using sparsification methods, attribution tools, and research agents, and find three properties that make neurons interpretable units of computation:\n\nFirst, they are already functionally sparse. Ablating the 92.2% weakest MoE neuron activations for each forward pass has only modest effects on model capabilities – a 2.9% increase in language model loss.\n\nSecond, neurons have relatively few semantic groupings (modes) of computational functions, and we can identify these and describe them concisely. The majority of neurons have either one mode only or a single mode that is overwhelmingly causally significant. In cases where multiple causal modes are present, the number is small and legible to our methods.\n\nFinally, we can use agent-driven experimentation to generate neuron descriptions in natural language, and these descriptions allow us to predict, simulate, and activate the neuron. Providing the description to a judge model allows us to (1) predict whether the neuron will fire in a given passage of text with 95% accuracy vs random text; (2) simulate its token-by-token behavior in high-activation regimes sufficiently to recover 73% of the damage (as a proportion of a ceiling of damage restorable) from the neuron’s ablation; and (3) write text that causes the neurons to fire more strongly than anything observed in a 10B-token corpus.\n\nOverall, we argue that neurons are meaningful units of computation that can be effectively used for interpretability in their own right, letting us study the model on its own terms rather than through a learned proxy.\n\n## Introduction\n\nIn the [previous essay](https://already-clean.goodfire.com/) in this series, we argued that neurons in deep learning models are more interpretable and cleaner than has often been suggested. In this article, we describe what this means in more detail and share our empirical findings. Throughout, **the term “neuron” refers to an individual computational unit in the intermediate layer of a transformer MLP, whose output is a scalar activation, computed after the nonlinearity (SiLU), as explained in the Background section below.**\n\nNeurons were the first units through which deep networks were interpreted, and early work found them to be closely aligned with recognizable concepts ([Le et al., 2012](https://arxiv.org/abs/1112.6209); [Zeiler & Fergus, 2014](https://arxiv.org/abs/1311.2901); [Bau et al., 2017](https://arxiv.org/abs/1704.05796); [Olah et al., 2017](https://distill.pub/2017/feature-visualization/); [Radford et al., 2017](https://arxiv.org/abs/1704.01444); [Goh et al., 2021](https://distill.pub/2021/multimodal-neurons/)). However, this story was complicated by observations of so-called polysemanticity—the idea that individual neurons respond to seemingly unrelated inputs ([Nguyen et al., 2016](https://arxiv.org/abs/1602.03616); [Olah et al., 2020](https://distill.pub/2020/circuits/zoom-in/)). Polysemanticity is related to the proposal that more concepts are represented in a network than the number of neurons in the network, an idea canonized in the literature on “superposition” ([Elhage et al., 2022](https://transformer-circuits.pub/2022/toy_model/index.html); [Arora et al., 2018](https://arxiv.org/abs/1601.03764); [Park et al., 2023](https://arxiv.org/abs/2311.03658); [Costa et al., 2025](https://proceedings.neurips.cc/paper_files/paper/2025/hash/311bd32bcd952f93cf820bacd6955f0a-Abstract-Conference.html)). This, in turn, led to the development of learned overcomplete feature dictionaries in place of neurons as computational units of analysis ([Bricken et al., 2023](https://transformer-circuits.pub/2023/monosemantic-features/index.html); [Cunningham et al., 2023](https://arxiv.org/abs/2309.08600)). Despite these trends in the field, interpretability researchers continued to investigate respects in which neurons may still constitute a meaningful unit of analysis ([Geva et al., 2021](https://arxiv.org/abs/2012.14913); [Dai et al., 2022](https://arxiv.org/abs/2104.08696); [Geva et al., 2022](https://arxiv.org/abs/2203.14680); [Gurnee et al., 2024](https://arxiv.org/abs/2401.12181); [Arora & Wu, 2025](https://transluce.org/neuron-circuits); [Dravid et al., 2026](https://arxiv.org/abs/2606.03990); [Feucht et al., 2026](https://arxiv.org/abs/2605.01148); [Shabalin & Piotrowski, 2026](https://arxiv.org/abs/2610.24521)). We continue this line of work, introducing new ways to factor and understand what individual neurons compute.\n\nWhether a neuron is labeled monosemantic or polysemantic can depend on choices about what constitutes a single concept, and responses that initially appear unrelated may share a regularity that becomes apparent with a better understanding of the computations they support ([Murphy & Medin, 1985](https://doi.org/10.1037/0033-295X.92.3.289)). **We therefore characterize neurons by their computational functions as our primary object of study, considering semantic similarity of the texts that activate them as a secondary question.** We study the circuits in which each neuron participates, tracing how upstream neurons drive its activation and how downstream neurons use its output. This lets us examine how a neuron’s role varies across contexts and assess whether those roles admit a concise description.\n\nA common view is that neurons are an intractable unit of analysis. In the worst case, an investigation could have demonstrated this in three ways: (1) Individual neurons might rarely play identifiable computational roles, so that meaningful units would be clusters of neurons, or directions and manifolds in activation space; (2) Each neuron might take extreme values on a wide and heterogeneous range of inputs that no short description unifies; (3) Or neurons might resist descriptions precise enough to predict when, and how strongly, they fire. We test each possibility directly: we ablate weak activations, determine the semantic range of neurons’ computational roles, and predict and simulate activations from descriptions alone.\n\n**Our results provide evidence that neurons are tractable units of analysis.** Neurons (1) exert most of their downstream influence in high-activation regimes (above the 97th percentile at the positive extreme and below the 3rd at the negative); (2) have a small number of computational roles, distinguished by their interactions with other components of the network; and (3) can be given concise descriptions that predict and simulate their activations. The three results share a pattern: the stronger a firing, the more the model depends on it and the more precisely we can describe it.\n\nThe picture that emerges is not that every neuron means one thing, but that a neuron’s important firings have a small set of computational functions, and a short description captures them. Some neurons are genuinely polysemantic, and some behavior remains unexplained, but so far little of it appears critical for the model’s computation.\n\n**We therefore find that neurons are a useful and interpretable basis for understanding LLMs.**\n\n*Excerpt (abstract and introduction). Read the full paper, figures and neuron atlas at the [original article](https://www.goodfire.ai/research/neurons-as-an-interpretable-basis).*\n","body_html":"<h1 id=\"neurons-as-an-interpretable-basis\">Neurons as an Interpretable Basis</h1>\n<p>Curt Tigges, Matthew Kowal, Thomas McGrath, Owen Lewis, Nick Cammarata · Goodfire · November 10, 2026</p>\n<h2 id=\"abstract\">Abstract</h2>\n<p>A companion to <a href=\"https://already-clean.goodfire.com/\" rel=\"nofollow ugc noopener\">Neural Networks Are Already Clean</a></p>\n<p>An intuitive way to interpret a neural network is to study its neurons; that is, the hidden units of its MLP layers. However, they have long appeared too complex and polysemantic for direct study. This view has led much of the field to focus on learned feature spaces over sets of neurons, which makes interpretability more challenging: we first need to transform the model before normal science can begin.</p>\n<p>Building on a growing body of literature, our work suggests that neurons are more interpretable than previously assumed. We study the neurons of Qwen3.6-35B-A3B using sparsification methods, attribution tools, and research agents, and find three properties that make neurons interpretable units of computation:</p>\n<p>First, they are already functionally sparse. Ablating the 92.2% weakest MoE neuron activations for each forward pass has only modest effects on model capabilities – a 2.9% increase in language model loss.</p>\n<p>Second, neurons have relatively few semantic groupings (modes) of computational functions, and we can identify these and describe them concisely. The majority of neurons have either one mode only or a single mode that is overwhelmingly causally significant. In cases where multiple causal modes are present, the number is small and legible to our methods.</p>\n<p>Finally, we can use agent-driven experimentation to generate neuron descriptions in natural language, and these descriptions allow us to predict, simulate, and activate the neuron. Providing the description to a judge model allows us to (1) predict whether the neuron will fire in a given passage of text with 95% accuracy vs random text; (2) simulate its token-by-token behavior in high-activation regimes sufficiently to recover 73% of the damage (as a proportion of a ceiling of damage restorable) from the neuron’s ablation; and (3) write text that causes the neurons to fire more strongly than anything observed in a 10B-token corpus.</p>\n<p>Overall, we argue that neurons are meaningful units of computation that can be effectively used for interpretability in their own right, letting us study the model on its own terms rather than through a learned proxy.</p>\n<h2 id=\"introduction\">Introduction</h2>\n<p>In the <a href=\"https://already-clean.goodfire.com/\" rel=\"nofollow ugc noopener\">previous essay</a> in this series, we argued that neurons in deep learning models are more interpretable and cleaner than has often been suggested. In this article, we describe what this means in more detail and share our empirical findings. Throughout, <strong>the term “neuron” refers to an individual computational unit in the intermediate layer of a transformer MLP, whose output is a scalar activation, computed after the nonlinearity (SiLU), as explained in the Background section below.</strong></p>\n<p>Neurons were the first units through which deep networks were interpreted, and early work found them to be closely aligned with recognizable concepts (<a href=\"https://arxiv.org/abs/1112.6209\" rel=\"nofollow ugc noopener\">Le et al., 2012</a>; <a href=\"https://arxiv.org/abs/1311.2901\" rel=\"nofollow ugc noopener\">Zeiler &amp; Fergus, 2014</a>; <a href=\"https://arxiv.org/abs/1704.05796\" rel=\"nofollow ugc noopener\">Bau et al., 2017</a>; <a href=\"https://distill.pub/2017/feature-visualization/\" rel=\"nofollow ugc noopener\">Olah et al., 2017</a>; <a href=\"https://arxiv.org/abs/1704.01444\" rel=\"nofollow ugc noopener\">Radford et al., 2017</a>; <a href=\"https://distill.pub/2021/multimodal-neurons/\" rel=\"nofollow ugc noopener\">Goh et al., 2021</a>). However, this story was complicated by observations of so-called polysemanticity—the idea that individual neurons respond to seemingly unrelated inputs (<a href=\"https://arxiv.org/abs/1602.03616\" rel=\"nofollow ugc noopener\">Nguyen et al., 2016</a>; <a href=\"https://distill.pub/2020/circuits/zoom-in/\" rel=\"nofollow ugc noopener\">Olah et al., 2020</a>). Polysemanticity is related to the proposal that more concepts are represented in a network than the number of neurons in the network, an idea canonized in the literature on “superposition” (<a href=\"https://transformer-circuits.pub/2022/toy_model/index.html\" rel=\"nofollow ugc noopener\">Elhage et al., 2022</a>; <a href=\"https://arxiv.org/abs/1601.03764\" rel=\"nofollow ugc noopener\">Arora et al., 2018</a>; <a href=\"https://arxiv.org/abs/2311.03658\" rel=\"nofollow ugc noopener\">Park et al., 2023</a>; <a href=\"https://proceedings.neurips.cc/paper_files/paper/2025/hash/311bd32bcd952f93cf820bacd6955f0a-Abstract-Conference.html\" rel=\"nofollow ugc noopener\">Costa et al., 2025</a>). This, in turn, led to the development of learned overcomplete feature dictionaries in place of neurons as computational units of analysis (<a href=\"https://transformer-circuits.pub/2023/monosemantic-features/index.html\" rel=\"nofollow ugc noopener\">Bricken et al., 2023</a>; <a href=\"https://arxiv.org/abs/2309.08600\" rel=\"nofollow ugc noopener\">Cunningham et al., 2023</a>). Despite these trends in the field, interpretability researchers continued to investigate respects in which neurons may still constitute a meaningful unit of analysis (<a href=\"https://arxiv.org/abs/2012.14913\" rel=\"nofollow ugc noopener\">Geva et al., 2021</a>; <a href=\"https://arxiv.org/abs/2104.08696\" rel=\"nofollow ugc noopener\">Dai et al., 2022</a>; <a href=\"https://arxiv.org/abs/2203.14680\" rel=\"nofollow ugc noopener\">Geva et al., 2022</a>; <a href=\"https://arxiv.org/abs/2401.12181\" rel=\"nofollow ugc noopener\">Gurnee et al., 2024</a>; <a href=\"https://transluce.org/neuron-circuits\" rel=\"nofollow ugc noopener\">Arora &amp; Wu, 2025</a>; <a href=\"https://arxiv.org/abs/2606.03990\" rel=\"nofollow ugc noopener\">Dravid et al., 2026</a>; <a href=\"https://arxiv.org/abs/2605.01148\" rel=\"nofollow ugc noopener\">Feucht et al., 2026</a>; <a href=\"https://arxiv.org/abs/2610.24521\" rel=\"nofollow ugc noopener\">Shabalin &amp; Piotrowski, 2026</a>). We continue this line of work, introducing new ways to factor and understand what individual neurons compute.</p>\n<p>Whether a neuron is labeled monosemantic or polysemantic can depend on choices about what constitutes a single concept, and responses that initially appear unrelated may share a regularity that becomes apparent with a better understanding of the computations they support (<a href=\"https://doi.org/10.1037/0033-295X.92.3.289\" rel=\"nofollow ugc noopener\">Murphy &amp; Medin, 1985</a>). <strong>We therefore characterize neurons by their computational functions as our primary object of study, considering semantic similarity of the texts that activate them as a secondary question.</strong> We study the circuits in which each neuron participates, tracing how upstream neurons drive its activation and how downstream neurons use its output. This lets us examine how a neuron’s role varies across contexts and assess whether those roles admit a concise description.</p>\n<p>A common view is that neurons are an intractable unit of analysis. In the worst case, an investigation could have demonstrated this in three ways: (1) Individual neurons might rarely play identifiable computational roles, so that meaningful units would be clusters of neurons, or directions and manifolds in activation space; (2) Each neuron might take extreme values on a wide and heterogeneous range of inputs that no short description unifies; (3) Or neurons might resist descriptions precise enough to predict when, and how strongly, they fire. We test each possibility directly: we ablate weak activations, determine the semantic range of neurons’ computational roles, and predict and simulate activations from descriptions alone.</p>\n<p><strong>Our results provide evidence that neurons are tractable units of analysis.</strong> Neurons (1) exert most of their downstream influence in high-activation regimes (above the 97th percentile at the positive extreme and below the 3rd at the negative); (2) have a small number of computational roles, distinguished by their interactions with other components of the network; and (3) can be given concise descriptions that predict and simulate their activations. The three results share a pattern: the stronger a firing, the more the model depends on it and the more precisely we can describe it.</p>\n<p>The picture that emerges is not that every neuron means one thing, but that a neuron’s important firings have a small set of computational functions, and a short description captures them. Some neurons are genuinely polysemantic, and some behavior remains unexplained, but so far little of it appears critical for the model’s computation.</p>\n<p><strong>We therefore find that neurons are a useful and interpretable basis for understanding LLMs.</strong></p>\n<p><em>Excerpt (abstract and introduction). Read the full paper, figures and neuron atlas at the <a href=\"https://www.goodfire.ai/research/neurons-as-an-interpretable-basis\" rel=\"nofollow ugc noopener\">original article</a>.</em></p>","headings":[{"level":1,"text":"Neurons as an Interpretable Basis","id":"neurons-as-an-interpretable-basis"},{"level":2,"text":"Abstract","id":"abstract"},{"level":2,"text":"Introduction","id":"introduction"}]}}