Zubnet AILearnWiki › Cross-Attention
Fundamentals

Cross-Attention

Encoder-Decoder Attention
An attention mechanism where the queries come from one sequence and the keys/values come from a different sequence. In encoder-decoder models, the decoder's queries attend to the encoder's keys and values, allowing the decoder to "look at" the input while generating the output. Cross-attention is also how text conditions image generation in diffusion models — the image generation process attends to the text prompt.

Why it matters

Cross-attention is the bridge between different modalities and different parts of an architecture. It's how translation models connect source and target languages, how image generators follow text prompts, how multimodal models relate images to text, and how Retrieval-Augmented systems incorporate retrieved documents. Any time two different inputs need to interact, cross-attention is usually involved.

Deep Dive

In self-attention, Q, K, and V all come from the same sequence — each token attends to other tokens in the same input. In cross-attention, Q comes from one source (e.g., the decoder) and K, V come from another (e.g., the encoder). The decoder token asks "what in the input is relevant to what I'm generating right now?" and the attention mechanism provides a weighted summary of the input.

In Diffusion Models

Text-to-image models use cross-attention to condition image generation on text. The text prompt is encoded into embeddings (via CLIP or T5), and at each denoising step, the image features attend to the text embeddings through cross-attention layers. This is how the model knows to generate a "cat on a surfboard" — each spatial location in the image attends to the relevant words. Manipulating these cross-attention maps is how techniques like prompt weighting and attention editing work.

Attention Patterns

Self-attention and cross-attention have different computational profiles. Self-attention is quadratic in the sequence length (every token attends to every other token). Cross-attention is linear in the decoder length times the encoder length (each decoder token attends to all encoder tokens). In practice, the encoder output is often much shorter than the decoder sequence, making cross-attention cheaper than decoder self-attention.

Related Concepts

← All Terms
← Cosine Similarity Curriculum Learning →