Diffusion LLM
Why it matters
Deep Dive
An autoregressive model writes the way a person types: one token after another, each conditioned on everything before it, with no way to revise what is already down. A diffusion LLM works more like an editor with a full draft. Training teaches it to take a sequence where some or all tokens have been masked and predict the originals, so at inference time it can start from a completely masked canvas of the desired length and fill it in over a series of passes, unmasking or correcting tokens until the text converges. Because every position is updated in the same forward pass, the model can plan the whole answer at once and use context from both sides of each token. The idea is a direct transplant of the denoising process behind image diffusion models, reworked for a discrete vocabulary instead of continuous pixels.
How the Denoising Loop Works
Most text diffusion models use a masked (absorbing-state) formulation rather than the Gaussian noise used for images. During training, a random fraction of tokens in a clean sequence is replaced with a special mask token, and the model learns to predict what was hidden; the corruption rate varies from a few percent to the entire sequence, in a scaled-up cousin of the masked language modeling objective used to train encoder models. At generation time the model starts from a fully masked canvas, predicts a candidate token for every masked position in parallel, then commits only the tokens it is most confident about and re-masks the rest for the next step. After a few dozen such steps the sequence is complete. A common practical variant is semi-autoregressive decoding: the canvas is split into blocks that are generated left to right, but each block is filled in by parallel denoising, which blends diffusion's parallelism with some of the ordering guarantees of classic generation.
Why the Speed Math Is Different
An autoregressive model needs one forward pass per output token, so a 1,000-token answer means 1,000 sequential steps that cannot be parallelized; the KV cache keeps each step cheap, but the steps still happen one after another. A diffusion LLM needs one forward pass per refinement step, and the step count is a tunable dial that does not grow with output length, so in principle a short answer and a long answer cost similar latency. Inception Labs, the startup behind the Mercury models, advertises generation speeds above 1,000 tokens per second on standard GPUs — a number that comes from this parallelism rather than from exotic hardware. The tradeoff is that each pass recomputes attention over the whole canvas and gains little from caching, so the compute per pass is higher, and the real-world advantage shows up most clearly in throughput-oriented serving, where many sequences can be refined in a batch.
Bidirectional Context and Infilling
Because there is no left-to-right arrow in the architecture, a diffusion LLM conditions every token on everything around it, before and after. That makes infilling a first-class operation rather than a special mode: you can hand the model a document with a masked-out middle and ask it to regenerate just that span, keep the surrounding text fixed, or iteratively revise its own draft. This lineage goes back to encoder models like BERT, which showed that masked prediction learns strong bidirectional representations but was never used for open-ended generation; diffusion LLMs are in many ways the missing generative half of that idea. The same property helps with tasks where the beginning and end constrain the middle, such as completing a function body or rewriting a sentence to fit a fixed template.
It Won't Replace Autoregressive Models Yet
The demo numbers are real, but the gap to frontier autoregressive models has not closed. On standard evaluations, the best diffusion models land near strong open models of similar size rather than at the top of the leaderboard, and the tricks that push quality up — more refinement steps, heavier re-masking schedules — eat directly into the speed advantage. The harder structural problem is length: the model commits to a canvas size before it knows what the answer is, so it has to guess how long the response should be, pad short answers, or shrink and regrow the canvas mid-generation, all of which are active research areas rather than solved engineering. The tooling gap matters too: the serving stack, from vLLM to the major inference APIs, is built around left-to-right decoding with a key-value cache, so diffusion models currently run on bespoke infrastructure that most teams cannot simply adopt.
Who Is Building Them
Three names anchor the field. LLaDA, from academic researchers, showed that a masked diffusion model trained from scratch at the multi-billion-parameter scale can hold its own against comparable autoregressive models on instruction-following benchmarks, and its release as open weights gave the community a base to experiment on. Mercury, from Inception Labs, is the commercial flag-bearer: a family of API-served models focused on code and general chat that leans hard on the speed story. Gemini Diffusion is Google's experimental entry, positioned as a fast-sampling mode alongside its mainline models. Under the hood, much of the architecture work draws on the diffusion transformer research that modernized image generation, and most teams in the space treat diffusion not as a replacement for autoregressive modeling but as a second decoding paradigm to keep in the toolbox.