Machine Unlearning
Why it matters
Several pressures make this a practical need:
- Privacy law: rules such as the GDPR's "right to erasure" let people demand deletion of their personal data. Whether that reaches into trained models is still being worked out, but developers want an answer ready.
- Copyright: a court or settlement may require removing protected works from a model.
- Safety: labs want to strip out hazardous knowledge, such as details that would help in building biological weapons.
- Repair: removing poisoned, mislabeled or outdated data after the fact.
Deep Dive
Deleting a row from a database is easy. Deleting what a neural network learned from that row is not, because the information is smeared across billions of weights and mixed with everything else. The only sure way to remove it is to retrain the model without the data, which for a large model can cost millions of dollars.
Machine unlearning is the search for a cheaper path: modify the existing model so it behaves as if it had never seen the data.
Exact Versus Approximate
Exact unlearning guarantees the result matches retraining. The best-known approach, SISA, splits the training data into shards and trains a separate sub-model on each, so deleting a data point means retraining only one shard. It works for smaller models but doesn't fit the way LLMs are trained.
Approximate unlearning edits the trained model directly. Common methods:
- Gradient ascent: run training in reverse on the data to forget, raising the loss on it, usually balanced with normal training on data to keep.
- Substitute targets: fine-tune the model to give generic or "I don't know" answers on the forgotten topic. Microsoft researchers used a version of this in 2023 to make a Llama model largely forget the Harry Potter books.
- Representation methods: scramble the model's internal activations on the target topic so it can't reason about it, as in the RMU method.
Does It Really Forget?
This is the weak point. Research keeps finding that approximate unlearning often hides knowledge instead of erasing it:
- A small amount of fine-tuning on related data can bring the "forgotten" knowledge back.
- Jailbreaks and rephrased questions can extract it.
- One 2024 study found that simply quantizing an unlearned model restored much of what it was supposed to have lost.
- Unlearning also causes collateral damage: the model gets worse at nearby topics it was meant to keep.
Benchmarks such as TOFU (fictional author profiles), WMDP (hazardous knowledge) and MUSE test both forgetting and retained ability, and no method scores well on everything.
Unlearning Versus Related Ideas
- Knowledge editing changes a specific fact ("the CEO is now X"). Unlearning removes.
- Differential privacy limits what a model can memorize about any individual during training, preventing the problem up front.
- Output filters and guardrails block the model from saying something without changing what it knows.
Practical Advice
If data may need deleting later, the most reliable design keeps it out of the weights. Filter training data before training, and serve changeable or sensitive information through RAG, where deleting a document really deletes it. Treat today's unlearning methods as risk reduction, not as a guarantee, and test them adversarially before relying on them.