Six (and a half) intuitions for KL divergence
Summary
This article compiles six intuitive perspectives on KL divergence, including expected surprise, hypothesis testing, maximum likelihood estimation, suboptimal coding, and two gambling themed viewpoints, plus an exploration of Bregman divergence. It emphasizes that KL divergence measures how far a model Q is from the true distribution P in the context where P is true, explaining its asymmetry. It also connects KL to cross entropy and coding efficiency and discusses practical implications for learning and modeling.