NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute
Summary
NanoGPT Slowrun reports 10x data efficiency by training an ensemble of 1.8B parameter models on 100M tokens. The approach combines ensemble methods, chain distillation, regularization, looping, and architectural tweaks to push data efficiency and shift scaling leverage from data to compute.