The computation: for input X, compute Q = X·W_Q, K = X·W_K, V = X·W_V. Then: Attention(Q,K,V) = softmax(Q·K^T / √d_k) · V. The softmax(Q·K^T) produces an N×N attention matrix where entry (i,j) represents how much token i attends to token j. The √d_k scaling prevents dot products from growing too large in high dimensions, which would push softmax into saturated regions with near-zero gradients.
In decoder-only LLMs (GPT, Claude, Llama), self-attention is causal: each token can only attend to previous tokens (including itself). This is enforced by a causal mask that sets future positions to −∞ before softmax. In encoder models (BERT), self-attention is bidirectional: every token attends to every other token. The causal constraint is what makes autoregressive generation possible — the model can't "peek" at future tokens.
Self-attention computes an N×N attention matrix, making it O(N²) in both time and memory. For a 128K token context, that's ~16 billion entries per layer per head. This quadratic scaling is the fundamental limitation that drives research into sparse attention, linear attention, Flash Attention (which reduces memory but not compute), and SSMs (which avoid the N×N matrix entirely). Every approach to long-context modeling is ultimately about managing this quadratic cost.