Understanding FlashAttention Pt 1: Personal Notes
Summary
Understanding FlashAttention Pt 1: Personal Notes provides a deep dive into IO-aware attention, focusing on tiling, online softmax, and recomputation to reduce GPU memory traffic while preserving exact dense attention. The post covers the fundamental problem, the mathematical trick behind streaming softmax, GPU implementation considerations, and how these ideas translate into forward and backward passes, with practical notes and an educational PyTorch example.