Data Poisoning
Why it matters
LLMs are trained on huge amounts of text scraped from the public web, and anyone can publish to the web. That means anyone could, in principle, try to influence what future models learn. Research has shown this is more practical than once assumed:
- A 2023 study showed attackers could poison popular web-scale image datasets for about $60 by buying expired domains that the datasets still pointed to.
- A 2025 study by Anthropic, the UK AI Security Institute and the Alan Turing Institute found that as few as 250 poisoned documents could plant a backdoor in models from 600 million to 13 billion parameters. The number needed stayed roughly constant as models and datasets grew, rather than scaling with dataset size.
Fine-tuning data, RAG knowledge bases and shared open models are also targets, so builders face this risk too, not only big labs.
Deep Dive
A model is only as trustworthy as the data it learns from. Data poisoning exploits that: an attacker adds carefully crafted examples to the training data so the finished model behaves the way the attacker wants. Sometimes the goal is to make the model worse overall. More often, it's a backdoor: the model behaves normally until it sees a specific trigger, such as a rare phrase, and then does something else, like producing harmful content, leaking data or writing insecure code.
Because the trigger is secret, a backdoored model can pass ordinary testing perfectly.
Kinds of Poisoning
- Backdoors: a trigger causes a specific behavior. Anthropic's 2024 "Sleeper Agents" research showed backdoored behaviors can survive standard safety training.
- Targeted poisoning: make the model wrong about one specific thing, such as a brand, a person or a piece of code.
- Availability attacks: degrade the model's quality in general.
- Artist-led poisoning: tools such as Nightshade subtly alter images so that image models that train on them without permission learn wrong associations. It's poisoning used as a form of protest or protection.
Beyond Pre-training
- Fine-tuning datasets: a small number of bad examples in a shared dataset can teach a model harmful habits.
- RAG poisoning: planting documents in a knowledge base or on the web that a retrieval system will pull in. It's closely related to indirect prompt injection.
- Model supply chain: a downloaded model or adapter may already be backdoored. Use trusted sources and safe formats like safetensors.
Defenses
- Data provenance: know where training data comes from, and prefer curated, trusted sources.
- Filtering and deduplication: remove low-quality, repeated or anomalous content before training.
- Evaluation for triggers: red-team models with unusual inputs; specialized detection methods look for backdoor signatures.
- For RAG: control who can write to the knowledge base, and treat retrieved content as untrusted input.
- For fine-tuning: review datasets, especially community-contributed ones, and test the fine-tuned model against your safety checks before deploying it.