Pretraining Language Models via Neural Cellular Automata
Summary
A research blog proposes training language models using Neural Cellular Automata (NCA) as synthetic data instead of relying on large text corpora. It reports that 164M NCA tokens can outperform traditional pretraining methods on multiple domains, with faster convergence and better final perplexity, and argues that structure and in-context rule inference, not semantics, drive transfer. The work also emphasizes tuning data complexity to domain and envisions foundation models trained on synthetic data before language data.