The core challenge: RoPE (Rotary Position Embeddings) encodes position using rotation angles. At positions beyond the training length, these angles become extrapolations that the model has never seen, causing attention patterns to break down. Extension techniques modify how positions map to rotation angles so that longer sequences produce angles within the model's trained range.
NTK-aware interpolation (Neural Tangent Kernel) adjusts RoPE frequencies non-uniformly: high-frequency components (important for local patterns) are preserved while low-frequency components (position-dependent) are interpolated. This preserves the model's ability to handle local patterns (word order, syntax) while extending its range for global position encoding. It's a one-line code change that dramatically improves length extrapolation.
YaRN (Yet another RoPE extensioN) combines NTK-aware interpolation with an attention temperature correction and a small amount of fine-tuning on extended-length data (typically a few hundred steps). This produces models that handle 4–8x their original context length with minimal quality degradation. Most open-source long-context models (like long-context Llama or Mistral variants) use YaRN or similar techniques. The fine-tuning step is crucial — scaling alone works somewhat, but fine-tuning at the target length significantly improves quality.