Skip to main content
Zubnet AI › Learn › Wiki › Adversarial Examples
Safety

Adversarial Examples

Also known as: Adversarial Attack, Adversarial Perturbation
Inputs deliberately crafted, often with changes invisible to humans, to make an AI model give a wrong or harmful output.

Why it matters

Adversarial examples showed that a model's high accuracy on normal data says little about how it behaves when someone is actively trying to fool it. That's crucial wherever AI makes security- or safety-relevant decisions: self-driving cars reading road signs, face recognition, malware detection, content moderation and fraud detection.

They aren't limited to lab settings. Researchers fooled sign classifiers with a few stickers on a stop sign, made a 3D-printed turtle that was classified as a rifle from almost any angle, and designed printed patches that could hide a person from object detectors.

With LLMs, the same idea reappears as adversarial prompts: strings of text optimized to break a model's safety training.

Deep Dive

In 2013, researchers discovered something unsettling about neural networks: you could take an image the model classified correctly, change its pixels by an amount too small for a person to notice, and make the model confidently get it wrong. A famous 2014 example showed a picture of a panda, plus a faint layer of carefully computed noise, classified as a gibbon with 99% confidence. To a human, both images looked identical.

These crafted inputs are called adversarial examples. They work because models rely on patterns that humans don't perceive, and an attacker can push on exactly those patterns.

How Attacks Are Made

  • White-box attacks: if you have the model's weights, you can compute the gradient of its output with respect to the input and nudge the input in the direction that most increases the error. The Fast Gradient Sign Method (FGSM) does this in one step; stronger methods like PGD take many small steps.
  • Black-box attacks: without the weights, attackers query the model repeatedly, or craft examples on a similar model they control. Adversarial examples often transfer: one that fools one model frequently fools others too.
  • Physical attacks: perturbations designed to survive printing, lighting changes and different viewing angles.

Adversarial Attacks on LLMs

In 2023, researchers showed an automated method (often called GCG) that searches for a short suffix of gibberish-looking tokens which, appended to a harmful request, makes aligned models comply. Suffixes found on open models also worked on some closed models. Since then, attacks have ranged from optimized tokens to images with hidden instructions for multimodal models. This overlaps with jailbreaking and prompt injection, though those are often written by hand rather than computed.

Defenses

  • Adversarial training: include adversarial examples in training so the model learns to resist them. The most effective general method, but costly, and it can reduce accuracy on clean inputs.
  • Certified robustness: mathematical guarantees that no change within a small range can flip the output. Strong, but only practical for small changes and smaller models.
  • Input preprocessing and detection: filtering or transforming inputs to destroy perturbations. Many such defenses were later broken by adaptive attacks.
  • System design: don't let a single model's output be the only safeguard for high-stakes decisions; combine it with other checks.

The Big Lesson

No widely used defense makes models fully robust. Adversarial examples are a reminder that models see the world differently from people, and that security means testing against a thinking adversary, not just against typical data.

← All Terms
ESC