Emulating ALiBi with RoPE
Summary
The article explains how to emulate ALiBi using RoPE for LLMs, detailing the construction that adds a fixed-rotation pair to Q and K, the role of RoPE frequencies, and how the small-angle limit can reproduce ALiBi. It discusses practical considerations and notes a BLOOM-560M test showing close results.