bf16 might be freezing your weight updates
Summary
A detailed exploration of bf16 precision impact on AdamW fine-tuning, showing that many weight updates are effectively dropped, leading to sparse training and a 'cliff' in recommended learning rates. The piece compares bf16-only training to FP32 master copies and discusses fixes like stochastic rounding and using a fp32 master copy, with experimental results and code references.