Mechanistic interpretability

Research that tries to reverse-engineer a neural network's internal computation by identifying its features and circuits.

Unlike methods that only explain a single prediction after the fact, mechanistic interpretability asks how the network actually computes its answer. Typical tools include probing, activation patching, ablation and sparse autoencoders. The field is young, and researchers say they understand only a small part of large models.

#Glossary #Interpretability

Stories about Mechanistic interpretability

Simplified diagram of a neural network whose middle-layer units respond to features such as a star shape or a ringed texture

InterpretabilityResearch

Interpretability explained: neurons, features, circuits and superposition

How researchers try to read the inside of neural networks, from single neurons to circuits and sparse autoencoders, and why the problem is still unsolved.

thelatentdaily.com Editorial TeamOct 224 min read