Particles Don’t Care About Z: Towards Scaling Entropy Estimation of Unnormalized Densities

Abstract

Computing the differential entropy of distributions known only up to a normalization constant is a long-standing problem. Variational inference scales well for approximating densities from samples but is underused when only the unnormalized density is available; the difficulty is building variational distributions that exploit the density’s structure, stay expressive, remain tractable, and sample efficiently. The paper shows that the earlier P-SVGD method (Messaoud et al., 2024) fails to scale to high dimensions due to several algorithmic flaws, including mishandled SVGD hyperparameter sensitivity, a violated global-invertibility assumption, a missing trace-of-Hessian term, and weak heuristics. It introduces MET-SVGD, which fixes these with a principled framework for SVGD hyperparameter selection plus invertibility and convergence guarantees, enabling accurate, scalable high-dimensional entropy estimation. Reported gains include up to 12× and 16× accuracy improvements over P-SVGD and SVGD baselines, an 80.4% FID improvement on CIFAR-10 energy-based generation with 64× better training stability, and up to 16% higher returns in Maximum-Entropy RL.