Architecture: the SAE takes a model's activation vector (dimension d_model, e.g., 4096) and encodes it into a much larger sparse representation (e.g., 64K features, of which only ~100 are active for any given input). It then decodes back to d_model and is trained to minimize reconstruction error. The sparsity constraint (L1 penalty on the hidden layer) forces the SAE to use only a few features per input, ensuring each feature is specific rather than diffuse.
When trained on LLM activations, SAEs discover interpretable features: a "Golden Gate Bridge" feature that activates on text about the bridge, a "Python code" feature, a "French language" feature, a "sycophantic agreement" feature, and so on. These features are more interpretable than individual neurons because the sparsity constraint separates overlapping concepts that neurons represent in superposition. Anthropic's research found features ranging from concrete (specific entities) to abstract (deception, uncertainty).
SAE features can be used for more than understanding: clamping a feature to zero suppresses the corresponding behavior (deactivating a "deception" feature), while amplifying a feature strengthens it. This opens the possibility of fine-grained behavioral control without retraining. However, the technique is still experimental — feature interactions are complex, and modifying one feature can have unintended effects on others due to residual superposition.