Exploding variance of means of exponentials: least-squares to the rescue
Summary
The article analyzes the problem of exploding variance when estimating log-sum-exp and KL divergence, introducing a spectral framework built from a continuum of least-squares problems to estimate relative density. It develops a variational formulation with two potentials, extends it via integration over a parameter, and derives closed-form spectral estimators using linear feature maps and eigen decompositions. The piece discusses computational advantages, potential for end-to-end feature learning, and implications for density estimation and neural network design.