Attention Residuals (AttnRes) – Depth-aware residuals for Transformers
Summary
Attention Residuals introduces AttnRes, a drop-in replacement for standard residual connections in Transformers, enabling each layer to attend over earlier representations via learned attention over depth. Block AttnRes reduces memory from O(Ld) to O(Nd) by partitioning layers into blocks, while preserving most of the gains. The repository provides formulas, PyTorch-style pseudocode, and benchmarking results showing improvements on multiple tasks, demonstrating depth-wise attention can improve both reasoning and code generation.