Researchers from Northwestern, Tilde Research, and University of Washington published Parallax (arxiv 2605.29157, code at github.com/Yifei-Zuo/Parallax), an attention mechanism that does the architectural opposite of what most linear-attention work does. Instead of replacing softmax to get sub-quadratic complexity, Parallax keeps softmax exactly and adds a learned covariance correction branch on top. The motivation is quality not speed: the underlying theory (Local Linear Attention, derived from nonparametric statistics in a test-time regression framework) upgrades softmax's "local constant estimate" of attention to a "local linear estimate," with provably better bias-variance tradeoffs for associative memory. At 1.7B scale with the Muon optimizer, Parallax beats the Transformer baseline by 1.02 points on downstream accuracy (62.45 vs 61.43), and tops Mamba, Gated DeltaNet, MesaNet, and Kimi DeltaAttention on MAD-Benchmark with 0.716 average accuracy.
The mechanism is additive. Output equals softmax attention minus a projected covariance term: the KV covariance gets multiplied by a learned probe ρᵢ = WR·xᵢ, where WR is a learnable projection matrix that probes the KV covariance directly from the layer input. The crucial design property: setting WR = 0 recovers standard softmax exactly. That means you can convert a pretrained Transformer checkpoint by adding WR and fine-tuning, without starting from scratch. Compute story: Parallax doubles arithmetic intensity by reusing the same KV stream for both branches, and on a NVIDIA H200 decode kernel matches or outperforms FlashAttention 2 and 3 (1.54x speedup compute-matched, 1.14x I/O-matched, across batches 1 to 2048 and contexts 128 to 32768). Critical caveat: gains depend on optimizer. Under Muon, the architecture has substantial advantage. Under AdamW, the advantage shrinks markedly or disappears, an unusual architecture-optimizer codesign dependency that limits where you can apply this drop-in. Important: this is NOT linear-complexity attention. KV cache is still required for decoding; the structure is softmax-attention-shaped, not Mamba-shaped.
Two threads worth pausing on. First, the narrative inversion is interesting on its own. The dominant linear-attention story for years has been "replace softmax to escape quadratic complexity," and Performer, Linformer, Hyena, Mamba, plus MiniMax M3's MSA all live in that frame. Parallax says: softmax was the local constant estimate of a more general thing, and the more general thing is the local linear estimate, which is theoretically better. The fix isn't to replace softmax, it's to extend it. That reframing matters for builders thinking about attention design: there's a layer of theory underneath the empirical landscape that suggests softmax isn't a ceiling, it's a starting point. Second, the optimizer-architecture codesign finding is honest and rare. Most papers don't publish "our gains evaporate under AdamW." Tilde Research and collaborators flagging this means future LLA-family work will need to track which optimizer regime it lives in, and Muon just gained a concrete reason to be studied alongside the architectural innovations it enables.
Monday morning, if you're training language models from scratch at the 1 to 2B scale and have Muon available: Parallax is worth treating as a drop-in baseline against vanilla softmax, given the open code at github.com/Yifei-Zuo/Parallax and the WR=0 backward-compat for pretrained-checkpoint conversion. The H200 decode kernel matches FlashAttention 2 and 3, so inference-side cost is not a barrier. If you're stuck on AdamW for compatibility reasons, the gains may not materialize; verify on your own training run before committing. If you're building from primitives and care about attention design as a first principle, the LLA theory (test-time regression, local linear estimate) is the paper to read; the Parallax implementation is one parameterization, not the only possible one. And if you're doing frontier-scale work, the 1.7B numbers are encouraging but the architecture-optimizer codesign needs verification at your scale before betting infrastructure.