Author: Francis Bach
\nPublished: September 25, 2026
\nSource: Machine Learning Research Blog
A common task in machine learning is to estimate or optimize “log-sum-exp” functions with (potentially continuously) many terms such as
\n[\\log \\Big( \\int_{\\mathcal{X}} e^{v(x)} dq(x) \\Big),]
\nwhere (v) is some potential and (q) a probability distribution. Applications include normalizing probabilistic models, smooth approximations to the maximum, transformers via derivatives, and entropy-regularized RL.
\nThe key difficulty is variance when (v) takes large values. For i.i.d. normals, the relative squared error for estimating (\\mathbb{E}[e^z]) is ((e^{\\sigma^2}-1)/n) — exploding exponentially in (\\sigma).
\nMain question: Can we keep the advantages of optimizing log-sum-exp while being less exposed to their computational/statistical disadvantages?
\nLeast-squares offers closed-form estimation for linear models, controlled variance via moments, and sharp analyses — but used naively for discrete outputs (e.g. one-hot classification) creates artefacts such as masking.
\nThe new attempt is summarized by the integral identity:
\n[t \\log t - t + 1 = \\int_0^1 \\frac{(t-1)^2}{\\rho t + 1-\\rho}(1-\\rho)\\, d\\rho.]
\nEstimate (\\log(dp/dq)), equivalent via variational formulations of KL / (f)-divergences (Nguyen–Wainwright–Jordan; Donsker–Varadhan). Empirical averages of exponentials are unstable; the post explores another way ([7]).
\nFor (f(t)=\\frac12\\frac{(t-1)^2}{\\rho t+1-\\rho}), the divergence has a quadratic variational form — exactly a least-squares prediction problem (related to noise-contrastive estimation).
\nWriting a general (f) as an integral of these weighted chi-squares yields a continuum of least-squares problems. KL corresponds to (d\\nu(\\rho)=2(1-\\rho)d\\rho). Potentials (v) and (w) are recovered by integrating the per-(\\rho) solutions.
\nWith linear features, each (\\rho) problem is a linear system. Integrating via a generalized eigenvalue decomposition of ((\\Sigma_p,\\Sigma_q)) yields a closed-form divergence and potentials — complexity (O(m^2n+m^3)), improvable with shared low-rank feature learning.
\nHigh-dimensional Gaussian analyses and simulations show spectral KL estimation reduces variance vs direct variational approaches when samples are limited; Pearson ((\\rho=0)) alone is geometrically less natural.
\nApplied to mutual information, the framework yields closed-form softmax-like conditional density estimation — not equivalent to one-hot least-squares — with large computational gains vs Newton softmax in some regimes.
\nMaximizing the spectral lower bound end-to-end with MM/EM-style algorithms enables feature learning at scale.
\nA continuum of stable least-squares problems, made feasible by one generalized eigendecomposition, circumvents exploding exponential means — with a closed-form path for softmax-style last layers. See arXiv:2605.10668 (NeurIPS).
\nFrontier LLMs used for figures/typos; technical content by the author.
","headings":[{"level":1,"text":"Exploding variance of means of exponentials: least-squares to the rescue","id":"exploding-variance-of-means-of-exponentials-least-squares-to-the"},{"level":2,"text":"The magic of least-squares","id":"the-magic-of-least-squares"},{"level":2,"text":"Relative density estimation as a testbed","id":"relative-density-estimation-as-a-testbed"},{"level":2,"text":"Weighted chi-square divergences","id":"weighted-chi-square-divergences"},{"level":2,"text":"Extension by integration","id":"extension-by-integration"},{"level":2,"text":"Linear models with closed-form spectral estimation","id":"linear-models-with-closed-form-spectral-estimation"},{"level":2,"text":"Benefits of spectral estimation","id":"benefits-of-spectral-estimation"},{"level":2,"text":"Mutual information and conditional estimation","id":"mutual-information-and-conditional-estimation"},{"level":2,"text":"Feature learning","id":"feature-learning"},{"level":2,"text":"Conclusion","id":"conclusion"}]}}