MLA
Why it matters
MLA is one of the reasons DeepSeek's models are cheap to serve. DeepSeek reported that it cut the KV cache by more than 90% compared with standard multi-head attention in DeepSeek-V2 while matching or beating its quality. The same design carried into DeepSeek-V3 and R1, the models that shook the industry in early 2025 with frontier-level results at low prices.
For anyone running models, a smaller cache means longer contexts, more simultaneous users per GPU and higher throughput. That makes MLA a direct competitor to GQA, which is still the default in most other open models.
Deep Dive
When a language model generates text, it keeps a KV cache: the keys and values of every previous token, for every layer, so it doesn't recompute them. For long contexts this cache becomes the main memory cost of serving a model. Multi-head Latent Attention, introduced by DeepSeek with DeepSeek-V2 in 2024, attacks that cost directly. Instead of caching full keys and values, it caches one small latent vector per token and rebuilds the keys and values from it when they're needed.
Earlier fixes, such as Multi-Query Attention and Grouped-Query Attention, saved memory by making heads share keys and values. MLA keeps every head distinct and compresses instead.
The Low-Rank Trick
In standard attention, each token's hidden state is projected into keys and values, one set per head. MLA first projects the hidden state down into a much smaller latent vector, and then projects that latent up into the keys and values for all heads. Only the small latent is cached. Mathematically, the key and value projections are forced to be low-rank, which works because keys and values across heads are highly redundant.
At inference time, the up-projection can be folded into neighboring matrices, so the model doesn't actually have to rebuild full keys and values for every step.
The Position Problem
Modern models encode token positions with RoPE, which rotates queries and keys depending on position. That clashes with compression: if position is baked into the keys, the cached latent would have to be recomputed as the sequence grows. MLA solves this with decoupled RoPE: a few extra dimensions of each query and a small shared key carry the positional information, while the compressed part stays position-free.
MLA Versus GQA and MQA
- MQA: all heads share one key/value head. Smallest cache among the sharing methods, but quality can suffer.
- GQA: heads share within groups. A middle ground, and the industry default.
- MLA: no sharing, compression instead. Cache size close to MQA with quality close to full attention, at the cost of a more complex design.
Adoption
MLA is more complicated to implement and optimize than GQA, and serving engines needed specific support for it. vLLM, SGLang and other major engines added that support as DeepSeek's models became popular. Research has also shown that existing GQA models can be converted to MLA-style attention after training. Whether MLA replaces GQA more broadly is still open, but it showed that the KV cache can be compressed far more than most people assumed.