Quadrillion-parameter transformers and how to train them
Summary
This article from Q Labs Research discusses scaling large language models under data-constrained regimes. It introduces a four-term scaling law for loss that adds an ensembling term and shows that larger models, longer training, and snapshot or deep ensembling can improve performance with more compute. It also presents practical prescriptions for training quadrillion-parameter networks and highlights extrapolation beyond Chinchilla baselines.