There Will Be a Scientific Theory of Deep Learning
Summary
The paper argues that a scientific theory for deep learning is emerging, aiming to characterize training dynamics, representations, final weights, and performance. It identifies five strands of work that collectively form a framework called learning mechanics, and discusses its relationship to mechanistic interpretability and other theoretical approaches.