Sliding window attention: each token attends only to tokens within a fixed window (e.g., 4096 tokens). Information from earlier tokens propagates through the layers — layer 1 sees 4096 tokens, layer 2 effectively sees 8192 (two windows worth), and by the final layer, information from the full sequence has had a chance to propagate. Mistral-7B uses a 4096-token sliding window across its 32 layers.
Longformer combines sliding window (local) attention with global attention on selected tokens (like [CLS] or user-defined positions). BigBird adds random attention connections on top of local and global patterns. These hybrid approaches let models handle 4K–16K tokens with subquadratic cost while maintaining the ability to connect distant tokens through global positions.
Sparse attention works well for many tasks but can degrade on tasks requiring precise long-range dependencies — referencing a specific fact from the beginning of a long document, maintaining consistency across a long conversation, or following complex instructions that span many tokens. Dense attention (full quadratic) with Flash Attention remains more robust for these cases, which is why most frontier models still use dense attention and rely on Flash Attention for efficiency rather than sparsity.