The original Transformer (2017) used fixed sinusoidal functions at different frequencies for each position and dimension. These had a nice theoretical property: the model could learn to attend to relative positions because the sinusoidal patterns create consistent offsets. But learned positional embeddings (a trainable vector for each position) quickly became the default because they performed slightly better, despite being limited to the maximum training length.
Rotary Position Embeddings (RoPE, Su et al., 2021) encode position by rotating the query and key vectors in the attention mechanism. The angle of rotation depends on position, so the dot product between two tokens naturally encodes their relative distance. RoPE is used by LLaMA, Mistral, Qwen, and most modern LLMs. Its key advantage: it enables length extrapolation — models can handle sequences somewhat longer than those seen during training, especially when combined with techniques like YaRN or NTK-aware scaling.
ALiBi (Attention with Linear Biases) takes a simpler approach: instead of modifying embeddings, it adds a linear penalty to attention scores based on distance between tokens. Farther tokens get penalized more. This requires no learned parameters and extrapolates well to longer sequences. Some architectures combine approaches or use relative position biases. The trend is toward methods that generalize beyond the training length, since context windows keep growing.