In-Context Learning
Why it matters
Deep Dive
Mechanically, in-context learning is just inference. The prompt — instructions, a few demonstrations, the new input — goes through the Transformer in a single forward pass, and the attention layers do the work: they match the new input against the demonstrations sitting in the context and produce a continuation consistent with the implied pattern. No gradients are computed, no weights are touched, and the model on disk is byte-identical before and after. What feels like learning is the model exploiting patterns it absorbed during pre-training about how tasks look when written down. The demonstration that made this famous was OpenAI's 175-billion-parameter GPT-3 paper, 'Language Models are Few-Shot Learners' (2020): one frozen model doing translation, question answering, and arithmetic from prompt text alone, at a level smaller models of the era could not reach.
Zero-Shot, One-Shot, and Few-Shot
The terms describe how many demonstrations the prompt contains. Zero-shot is instructions alone ('Classify this review as positive or negative'); one-shot adds a single worked example. Few-shot adds a handful — typically 2–10 for chat models, and up to a few dozen in research settings. A few-shot prompt for sentiment might show three short reviews, each followed by a label, then a fourth review and the word 'Sentiment:' — the model fills in the label, copying the format it was shown. Format consistency matters more than newcomers expect: same delimiter, same label vocabulary, same field order. Few-shot reliably beats zero-shot on classification, extraction, and formatting tasks where the output contract is hard to describe but easy to demonstrate.
The catch is that models are oddly sensitive to how the examples are picked and arranged. A well-documented failure mode is majority-label bias: if three of your four examples are labeled 'positive', the model leans positive regardless of the actual input. Recency bias (overweighting the last example) and example-order effects are also real — shuffling the same examples can swing accuracy by several points. Practitioners counter this with balanced label sets, hard representative examples rather than easy ones, and occasionally recalibrating output probabilities. This fragility is part of why the field pushed so hard on instruction tuning after 2020: a model that follows plain instructions needs fewer examples to babysit.
Why It Works at All
There is no settled explanation, but two theories dominate. The first comes from mechanistic interpretability: researchers reverse-engineering attention found circuits they call induction heads, which implement a simple algorithm — find the previous place in the context where the current pattern appeared, look at what came after it, and predict that again. That copy-the-pattern move is almost exactly what few-shot learning requires, and induction heads tend to form at roughly the point in training where in-context learning ability spikes, which is suggestive even if it is not the whole story. The second theory frames ICL as implicit Bayesian inference: pre-training text is a jumble of latent tasks and formats, and a coherent prompt lets the model infer which kind of document it is inside, then continue accordingly. The two views are compatible — one describes the circuit, the other the statistics — and both explain why clean, self-consistent prompts outperform sloppy ones.
What is not debated is the scale dependence. Few-shot ability is weak or absent in small models and improves sharply with parameters and training data, one of the cleanest examples of an emergent capability. This is why the GPT-3 result landed so hard: prompts that did nothing for a billion-parameter model worked on the 175-billion one. It is also why in-context learning quality is still a quiet differentiator between model generations — bigger models need fewer examples and tolerate messier ones.
Nothing Gets Saved
A persistent misconception is that the model 'learns' from your examples the way it would during training — that a good few-shot session improves the model for the next user. It does not. During inference the weights are frozen; the apparent learning is activation state inside the context window, and it evaporates the moment that context is dropped. Start a new chat and not a trace of your carefully chosen examples remains. That is the fundamental difference from fine-tuning, which rewrites weights with gradient descent and persists across every future request. If a chat product seems to remember you between sessions, that is a memory feature — stored notes re-injected into later prompts — not in-context learning.
The ephemerality has a cost: because nothing is saved, every request that needs the behavior must carry the examples again, and a 10-shot prompt with meaty examples can add thousands of tokens to each call. Prompt caching softens this — providers reuse the processed prefix at a discount — but for high-volume, narrow tasks a fine-tuned small model often wins on unit economics. The common rule of thumb: prototype with in-context learning, and reach for fine-tuning only when the prompt tax or the error rate gets embarrassing.
How Prompting Became the Interface
Before 2020, adapting a language model to a task meant transfer learning: take a pretrained model, fine-tune it on labeled data, deploy a specialist. GPT-3's few-shot results flipped that economics — if a frozen model could be programmed in natural language, one API could serve every task and users could bring their own. That inversion created prompt engineering as a discipline, made the system prompt a standard control surface, and set up later techniques that are really ICL in disguise: chain-of-thought prompting works partly by showing the model worked reasoning traces, and retrieval-augmented generation works by trusting the model to use whatever text lands in its context. The newer term 'context engineering' is the same insight grown up: the context window is the programming surface, and in-context learning is the interpreter that runs it.