Scaling Laws, Carefully
Summary
Scaling Laws, Carefully surveys the empirical scaling laws in deep learning, focusing on Kaplan et al.'s scaling laws for language models and the later Chinchilla findings. It discusses data-infinite versus data-limited regimes, the compute budget C ≈ 6ND, and the three methods used to fit scaling laws. The piece also covers reconciliations between Kaplan and Chinchilla and extensions addressing data repetition and overfitting penalties, while highlighting implementation caveats and the importance of robust experimental design.