Dispersion loss counteracts embedding condensation and improves generalization in small language models
Summary
The article presents embedding condensation as a geometric phenomenon in small language models where token embeddings collapse toward a narrow cone as layers deepen. It introduces dispersion loss to counteract this effect, provides experimental evidence across model sizes and datasets, and outlines how dispersion loss can improve generalization in smaller LMs without increasing parameters. The work includes theoretical framing, extensive figures, and practical implementation notes with citations and future directions.