What Rose Petals Teach Us about Induction
Summary
The article examines induction as a fundamental challenge in machine learning, using a dice-and-rose petals toy problem to compare how different algorithms encode inductive bias. It walks through Naïve Linear Regression, LR with categorical features, bincount pivots, fully connected neural nets, Deep Sets, and Decision Trees, highlighting how bias shape learning speed and generalization. It also connects these ideas to language models (LLMs) and broader AI research, arguing that no single general method for induction exists and that heuristics and architectural biases guide progress.