DigiNews

This article introduces a mechanistic interpretability framework for transformers, modeling them as a residual stream of linear projections and decomposing attention into separate OV (output-value) and QK (query-key) circuits. It analyzes one- and two-layer attention-only models to reveal how induction heads emerge via composition and how in-context learning may be implemented, offering methods like path expansion and ablations to validate the theory. The work provides a foundation for understanding transformer behavior in a way that could inform AI safety, research tooling, and future model analysis.