Understanding Positional Embeddings in Modern LLMs
Summary
This article explains positional embeddings in modern LLMs, comparing learned absolute position embeddings (GPT-2 style), fixed sinusoidal encodings, and RoPE (rotary position embeddings). It covers the mechanics of RoPE, relative vs absolute positioning, variations like partial RoPE and NoPE, and discusses architectural trends and practical considerations, with code examples and references.