The Training Example Lie Bracket
Summary
The article treats training examples as vector fields and derives the Lie bracket between two such fields to quantify how the order of presenting two examples affects SGD updates. It provides the mathematical framework, an experiment on a convnet trained with CelebA, and observations that Lie bracket magnitudes correlate with gradient magnitudes, with implications for diagnosing data-order effects in training.