Context Language Models
Summary
Context Language Models (CLMs) are proposed as language models that natively manage their own context by treating context as a writable file. The paper demonstrates that CLMs can be zero-shot trained to maintain and optimize context, enabling multi-agent contexts and improved performance across tasks. Additional contributions include an online reinforcement learning method for CLMs and a co-design of Suffix Cache Reuse to reduce server compute.