Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
Summary
The paper proposes Infinite-Parameter LLMs that generate and adapt model weights from live interaction data using a compact hypernetwork. Unlike fixed pretrained weights, online data modulates a shared base network, with a Bayesian belief over the latent code updated during sessions. This approach aims to persist knowledge across turns, reduce prompt reliance, and potentially outperform in-context learning and retrieval in adapting to live data.