Interpretability
Why it matters
People need to trust, debug and govern AI systems. A doctor reviewing an AI-assisted diagnosis, a bank explaining a credit decision to a regulator, or an engineer figuring out why a model fails on certain inputs all need some account of the model's reasoning. Laws increasingly ask for this: the EU AI Act sets transparency requirements for high-risk systems, and many financial and healthcare rules already require explanations for automated decisions.
For AI safety, interpretability is seen as one of the few ways to check what a model is actually doing, not just what it says. If a model could deceive its testers, looking at its outputs alone wouldn't be enough.
Deep Dive
Modern neural networks are often called black boxes. We know their inputs and outputs and can see every one of their billions of numbers, but we can't easily say why a model gave a particular answer. Interpretability is the umbrella term for methods that try to open the box.
It covers two broad traditions. Explainability (often called XAI) focuses on explaining individual decisions in human terms, such as which features mattered most for a loan rejection. Mechanistic interpretability, which has its own entry, tries to reverse-engineer the internal algorithms a model has learned.
Explaining Decisions
Classic explainability methods work from the outside:
- Feature attribution: methods such as SHAP and LIME estimate how much each input feature contributed to a prediction.
- Saliency maps: highlight which pixels in an image most influenced a classifier.
- Counterfactuals: "the loan would have been approved if income were 10% higher."
- Interpretable-by-design models: decision trees and linear models whose logic can be read directly, sometimes preferred in high-stakes settings even at some cost to accuracy.
Looking Inside
Methods for neural networks, and especially LLMs, look at internal activations:
- Probing: train a simple classifier on a model's hidden states to check whether it represents some concept, such as truthfulness or a sentence's sentiment.
- Sparse autoencoders: decompose activations into millions of interpretable "features". Anthropic's 2024 work on Claude found features for concepts from the Golden Gate Bridge to code bugs, and showed that turning a feature up changes the model's behavior.
- Circuit tracing: map how features connect across layers to produce an answer, for example showing a model planning a rhyme before writing the line.
- Steering: adding or removing directions in activation space to change behavior, which both tests understanding and offers a control tool.
Reading the Model's Reasoning
Reasoning models produce a chain of thought before answering. Reading it is a useful, if partial, window into their process. But research shows a chain of thought isn't always faithful: models sometimes give reasons that don't match what actually drove their answer. Treat it as evidence, not proof.
Limits
Explanations can be misleading if they're simplified too far, and attribution methods sometimes disagree with each other. Internal methods have made fast progress but still explain only a fraction of what large models compute. Interpretability is a growing toolkit, not yet a full understanding.