Deep learning as program synthesis
Summary
The post argues that deep learning may implement a tractable form of program synthesis—learning compositional, algorithm-like solutions rather than just memorizing data. It reviews mechanistic interpretability evidence (e.g., grokking and vision circuits) and connects these to a broader hypothesis that neural networks perform a type of search for simple programs that explain data. It also outlines the need for formalizing a space of programs and suggests mathematical directions (e.g., singular learning theory, algebraic geometry) to ground this view and explains implications for generalization, scaling, and convergence.