Weight Space Misfolding
Summary
The article introduces the concept of weight-space misfolding, where direct edits to a model's weights can persistently alter its behavior. It presents a threat model with categories like factual substitution, denial of legitimate help, and selective omission, illustrated through three case studies using a target and editor model. Defensive recommendations focus on controlling editing, maintaining patch history, testing behavior, and ensuring a recovery path to bound risk in deployed systems.