Zubnet AILearnWiki › Mechanistic Interpretability
Safety

Mechanistic Interpretability

Mech Interp, MI
A research approach that tries to understand what's happening inside neural networks at the level of individual neurons, circuits, and features — not just what the model outputs, but how it computes those outputs. The goal is to reverse-engineer the algorithms that neural networks learn, the way you'd reverse-engineer compiled software to understand its source code.

Why it matters

If we're going to trust AI with important decisions, we need to understand how it makes them. Mechanistic interpretability is the most rigorous attempt at this — not just asking "what did the model do?" but "what algorithm did it implement and why?" It's central to AI safety research, particularly at Anthropic, and is producing real results: researchers have identified circuits for indirect object identification, induction heads, and modular arithmetic inside Transformers.

Deep Dive

The field draws on a key observation: neural networks don't store information in individual neurons (usually). Instead, they use superposition — many features are encoded as directions in activation space, with individual neurons participating in many features simultaneously. A neuron that seems to respond to "the concept of water" might actually respond to a superposition of features related to liquids, transparency, flow, and specific contexts. Disentangling these superposed features is one of the field's central challenges.

Sparse Autoencoders

One of the most promising recent tools is the sparse autoencoder (SAE). You train an autoencoder to reconstruct a model's internal activations, but with a sparsity constraint that forces it to use only a few features at a time. The learned features often correspond to interpretable concepts — a feature for "code comments," one for "French text," one for "mathematical reasoning." Anthropic published influential work using SAEs to find interpretable features in Claude, identifying millions of features including ones for deception, specific concepts, and language patterns.

From Features to Circuits

Beyond individual features, mechanistic interpretability tries to trace circuits: how does information flow through the network to produce a specific behavior? For example, "induction heads" are two-attention-head circuits that implement in-context learning by pattern-matching: if the model sees "A B ... A" it predicts B. These circuits have been found in models from 2-layer toy Transformers to full-scale LLMs. Understanding circuits at scale remains an open challenge, but progress is accelerating.

Related Concepts

← All Terms
← MCP Memory →