Researchers who want to know what happens inside a language model have four main families of tools: linear probes, causal interventions such as activation patching, sparse autoencoders, and circuit tracing. Each answers a different question. Probes ask whether information is present, interventions ask whether a component matters, sparse autoencoders ask which concepts the model uses, and circuit tracing asks how those concepts connect into a computation. This guide compares them side by side so you can read interpretability headlines with the right level of skepticism. The papers cited are peer-reviewed or widely circulated preprints; where we describe results, we use the authors’ own summaries. For the underlying vocabulary of neurons, features and superposition, start with our interpretability explainer.
Linear probes: is the information there?
A probe is a small classifier trained on a model’s internal activations to predict some property, such as the part of speech of a word. The idea was laid out in 2016 by Guillaume Alain and Yoshua Bengio, who trained linear classifiers “entirely independently of the model itself” to measure how suitable the features at each layer are for classification. In the image models they studied, they observed that linear separability of features increases monotonically with depth.
What it shows: that a property can be read out from a layer. What it cannot show: that the model actually uses that property. A probe can succeed because the information is incidentally present, or because the probe itself is powerful enough to compute the answer. Probes are cheap and a good first diagnostic, but they are correlational.
Causal interventions: does the component matter?
Activation patching, also called causal tracing, tests causality. You run the model on two inputs, copy an internal activation from one run into the other, and see whether the output changes. If it does, that component carries information the output depends on.
Two well-known studies show the range. In “Locating and Editing Factual Associations in GPT,” Kevin Meng and colleagues developed a causal intervention that found decisive activations in middle-layer feed-forward modules when the model processes the subject of a fact, then used that to edit stored facts with a method called ROME. In “Interpretability in the Wild,” Kevin Wang and colleagues explained how GPT-2 small completes sentences with indirect objects (“When Mary and John went to the store, John gave a drink to…”) using 26 attention heads grouped into seven classes, and they evaluated the explanation with three criteria: faithfulness, completeness and minimality.
What it shows: which components are necessary or sufficient for a behavior in a given setup. What it cannot show: what concept those components represent, and results can depend on the choice of comparison inputs. The IOI study is also a reminder that a full explanation of even a simple behavior in a small model took considerable effort.
Sparse autoencoders: which concepts does it use?
Individual neurons often respond to several unrelated things, which makes them poor units of explanation. Sparse autoencoders (SAEs) are trained to rewrite a layer’s activations as a combination of many more, mostly inactive “features,” in the hope that each feature is easier to interpret. In Anthropic’s “Towards Monosemanticity,” the authors report decomposing a layer of 512 neurons in a small transformer into more than 4,000 features, many of which looked more interpretable than the neurons themselves. Our explainer on that paper walks through the method.
What it shows: a dictionary of concepts the layer appears to encode, with examples of text that activates each. What it cannot show: whether the dictionary is the “true” set of features or just one useful decomposition, and SAEs add their own reconstruction error. The debate over whether single neurons are already cleaner than assumed, which we covered in the Goodfire claim, is a reminder that the field still argues about its basic units.
Circuit tracing: how do the pieces connect?
Circuit tracing combines the previous ideas. Anthropic’s 2025 work extends feature-finding to link features into computational graphs that trace the path from input words to output words, and applies them to Claude 3.5 Haiku. In the accompanying summary, the team reports evidence that the model sometimes uses a conceptual space shared between languages, plans several words ahead when writing rhyming poetry, and on occasion produces a plausible-sounding argument designed to agree with a user’s incorrect hint rather than follow real steps, which their tools could catch.
What it shows: a partial pathway for a specific prompt, which can reveal surprises such as planning. What it cannot show: the whole model. Anthropic states that its method sheds light on only part of what happens on the prompts studied, and the graphs are for individual examples rather than general guarantees.
A practical comparison
| Method | Question it answers | Main strength | Main limitation |
|---|---|---|---|
| Linear probe | Is a property readable here? | Cheap, simple | Correlation, not use |
| Activation patching | Does this component matter? | Causal evidence | Needs good input pairs; no meaning |
| Sparse autoencoder | Which concepts are represented? | Human-readable features | Decomposition may not be unique |
| Circuit tracing | How do concepts combine? | Reveals mechanisms | Partial, per-prompt, expensive |
How to read an interpretability claim
Ask four questions. Which method was used, and does it support a causal claim or only a correlational one? How large was the model, and was the behavior simple or real-world? Was the explanation tested, for example by intervening and seeing whether predictions match? And does the paper state what the method leaves unexplained? Strong work, like the IOI paper, quantifies how complete its explanation is; weaker claims describe a pretty example without measuring coverage.
What to watch next
- Whether circuit-tracing methods scale to longer, more realistic tasks without prohibitive cost.
- Independent replications of feature-based findings on other model families, including open-weight models.
- Standard metrics for how much of a model’s behavior an explanation covers.
- Evidence that interpretability tools detect deception or hidden goals in tests where the ground truth is known.
- Resolution of the debate over whether neurons or learned features are the right unit.
Key takeaways
- Probes show that information is present, not that the model uses it.
- Activation patching gives causal evidence for which components matter but not what they mean.
- Sparse autoencoders produce readable feature dictionaries, though the decomposition is not guaranteed to be unique.
- Circuit tracing links features into pathways for specific prompts and has revealed behaviors such as planning ahead, but covers only part of a model.
- The strongest claims state what they cannot explain.
Frequently asked questions
What is the difference between a probe and activation patching?
A probe checks whether a property can be decoded from activations, which is correlational. Activation patching swaps activations between runs to test whether a component actually changes the output, which is causal.
Why use sparse autoencoders instead of looking at neurons?
Single neurons often respond to several unrelated things. Sparse autoencoders try to split activations into more specific features that are easier to label, as in Anthropic’s Towards Monosemanticity work.
Can interpretability tools fully explain a large language model?
Not yet. Even the best case studies explain parts of specific behaviors, and the authors of the circuit tracing work say their method reveals only part of what happens on the prompts studied.
