Dispersion loss counteracts embedding condensation and improves generalization in small language models
Summary
A recent paper investigates embedding condensation in small language models and introduces dispersion loss to counteract this geometry, improving generalization. The study shows condensation is stronger in smaller models, occurs early, and is not fixed by distillation, with dispersion loss offering a practical regularizer to align small LMs' representations with larger ones.