You Could Have Come Up With Kimi Delta Attention
Summary
Technical explainer of DeltaNet and Kimi Delta Attention (KDA), tracing the evolution from quadratic to linear attention, the delta-update mechanism, gated variants, and per-channel forgetting. The article also covers practical implications for implementations (Triton kernels) and chunk-wise processing for training vs decoding.