Skip to main content
Zubnet AI › Learn › Wiki › Multi-Query Attention
Models

Multi-Query Attention

Also known as: MQA
An attention design where all query heads share a single set of keys and values, shrinking the KV cache and speeding up inference.

Why it matters

During text generation, the bottleneck is usually not computing attention but loading the KV cache from memory for every new token. That cache grows with context length, number of layers and number of key/value heads. MQA cuts the head count to one, which shrinks the cache dramatically and lets a server fit longer contexts or many more users on the same GPU.

MQA was used in models such as Google's PaLM and the original Falcon. It also directly inspired Grouped-Query Attention (GQA), the compromise that most open models use today.

Deep Dive

In standard multi-head attention, every attention head has its own queries, keys and values. A model with 32 heads stores 32 sets of keys and values for every token in the context. Multi-Query Attention, proposed by Noam Shazeer in 2019, keeps the 32 separate query heads but makes them all share just one key head and one value head.

The model still looks at the context in 32 different ways, because the queries differ. But the memory it needs to remember the context drops by up to 32×.

The Memory Math

KV cache size is roughly: 2 (keys and values) × layers × KV heads × head dimension × sequence length × bytes per value. Take a model with 32 layers, 32 heads of dimension 128, in FP16, at 32K tokens of context. With full multi-head attention, the cache is about 16 GB per sequence. With MQA it's about 0.5 GB. That difference decides whether you can serve one long conversation per GPU or dozens.

The Quality Cost

Sharing one set of keys and values is a big constraint. Every head must find what it needs in the same representation, which can hurt quality, especially on harder tasks. MQA models also tended to be less stable during training. That's why the field largely moved to:

  • GQA (Grouped-Query Attention): query heads are split into groups, and each group shares a key/value head. With 32 query heads and 8 KV heads, you get a 4× cache saving with almost no quality loss. Llama 2 70B, Llama 3, Mistral and many others use it.
  • MLA (Multi-head Latent Attention): introduced by DeepSeek, it compresses keys and values into a small latent vector and expands them when needed, cutting the cache even further while keeping quality high.

MQA is really the extreme end of the GQA spectrum: GQA with one group.

Converting Existing Models

You don't have to train from scratch. The GQA paper showed you can "uptrain" a multi-head model into GQA or MQA by averaging its existing key and value heads into fewer heads and then training briefly (around 5% of the original pre-training compute) to recover quality.

When It Matters to You

If you serve models yourself, the number of KV heads (often listed as num_key_value_heads in the model config) tells you how memory-hungry long contexts will be. A model with num_key_value_heads: 1 is MQA; a value smaller than num_attention_heads is GQA.

← All Terms
ESC