The Hot Mess of AI: How Does Misalignment Scale with Model Intelligence and Task Complexity?
Summary
A study from the Alignment Science Blog decomposes AI errors into bias and variance to measure incoherence. It finds that longer reasoning increases incoherence, scale improves coherence on easy tasks but not hard ones, and ensembling can reduce variance; synthetic optimizer experiments show larger models reduce bias faster than variance. The authors argue future AI failures may resemble industrial accidents rather than coherent misalignment, urging a shift in alignment priorities toward guarding against reward hacking and goal misspecification in training, with implications for enterprise governance and risk management.