Language Models Need Sleep
Summary
This arXiv paper proposes a sleep-like consolidation mechanism for transformer-based LLMs, where recent context is converted into persistent fast weights during a 'sleep' phase. Offline recurrent passes update the fast weights in state-space model blocks, shifting computation to sleep while keeping wake-time latency intact. Results show performance gains on tasks requiring deeper reasoning, suggesting a path to improved long-horizon reasoning with reduced wake-time computation.