Convergent Evolution: How Different Language Models Learn Similar Number Representations
Summary
The paper shows that different language models learn periodic number representations with similar Fourier-domain features. It identifies a two-tiered hierarchy where all models exhibit Fourier-domain number features, but only a subset learn geometrically separable representations, and it explains how data, architecture, optimizer, and tokenizer influence this. The findings highlight convergent evolution in neural feature learning and implications for numerical reasoning in language models.