Smallest transformer that can add two 10-digit numbers
Summary
Overview of a GitHub project exploring the smallest autoregressive transformer capable of adding two 10-digit numbers with high accuracy. It presents two submission tracks (trained and hand-coded), a leaderboard of architectural tricks (rank-3 factorization, ALiBi, shared KV, RMSNorm), and a verification workflow using verify.py seeded to 2025. The work highlights how extreme model compression and creative tokenization can realize numerical arithmetic tasks with minimal parameters.