Adaptive Variational Inference in Probabilistic Graphical Models: Beyond Bethe, Tree-Reweighted, and Convex Free Energies

TL;DR

The paper addresses the reliability of variational inference in probabilistic graphical models by examining two generalizations of the Bethe free energy: Fc, which reshapes the entropy via counting numbers, and Fzeta, which scales the state energy. It analyzes how these classes behave on attractive and mixed graphs and introduces adaptive schemes ADAPT-$c$ (for mixed models) and ADAPT-$ ext{zeta}$ (for attractive models) that tailor the free-energy approximation to the specific model. Empirical results on complete, grid, and Erdos–Renyi graphs show that ADAPT-$c$ can drastically improve partition-function estimates while ADAPT-$ ext{zeta}$ enhances singleton marginals in densely connected attractive models; in many settings SBP and convex-free-energy variants remain competitive for marginals. Overall, the work provides practical, model-aware strategies to improve variational approximations beyond Bethe, TRW, and standard convex entropies, with clear implications for more robust probabilistic inference in complex graphs.

Abstract

Variational inference in probabilistic graphical models aims to approximate fundamental quantities such as marginal distributions and the partition function. Popular approaches are the Bethe approximation, tree-reweighted, and other types of convex free energies. These approximations are efficient but can fail if the model is complex and highly interactive. In this work, we analyze two classes of approximations that include the above methods as special cases: first, if the model parameters are changed; and second, if the entropy approximation is changed. We discuss benefits and drawbacks of either approach, and deduce from this analysis how a free energy approximation should ideally be constructed. Based on our observations, we propose approximations that automatically adapt to a given model and demonstrate their effectiveness for a range of difficult problems.

Figures (23)

• Figure 1: Local polytope $\mathbb{L}$ as the linearly constrained domain of the Bethe free energy $\mathcal{F}_B$ (simplified illustration on a single-edge graph and two nodes $X_1,X_2$).
• Figure 2: Approximation behavior of $\mathcal{F}_{\bm{c}}$. First row: attractive models; second row: mixed models.
• Figure 3: Approximation behavior of $\mathcal{F}_{\bm{\zeta}}$. First row: attractive models; second row: mixed models.
• Figure 4: Algorithms Bethe ($\mathcal{F}_B$), SBP, TRW, LS-Convex, and ADAPT-$c$'compared on attractive models. First row: $l^1$- error on singleton marginals; second row: $l^1$- error on pairwise marginals; third row: absolute error on log-partition function.
• Figure 5: Algorithms Bethe ($\mathcal{F}_B$), SBP, TRW, LS-Convex, and ADAPT-$c$ compared on mixed models. First row: $l^1$- error on singleton marginals; second row: $l^1$- error on pairwise marginals; third row: absolute error on log-partition function.