Human-like Neural Nets by Catapulting
Summary
Explores speculative method of building human-like NNs by catapulting through extreme overparameterization and high-learning-rate training to induce generalization. Links concepts like grokking, isoperimetry, and dynamic evaluation to discuss how such regimes could affect AI robustness, alignment, and scalability, and outlines potential experiments, datasets, and hardware considerations.