Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Guided Belief Propagation -- A Homotopy Continuation Method

Christian Knoll, Adrian Weller, Franz Pernkopf

TL;DR

This work enhances BP and proposes self-guided belief propagation (SBP) that incorporates the pairwise potentials only gradually and provides a formal analysis to demonstrate that SBP finds the global optimum of the Bethe approximation for attractive models where all variables favor the same state.

Abstract

Belief propagation (BP) is a popular method for performing probabilistic inference on graphical models. In this work, we enhance BP and propose self-guided belief propagation (SBP) that incorporates the pairwise potentials only gradually. This homotopy continuation method converges to a unique solution and increases the accuracy without increasing the computational burden. We provide a formal analysis to demonstrate that SBP finds the global optimum of the Bethe approximation for attractive models where all variables favor the same state. Moreover, we apply SBP to various graphs with random potentials and empirically show that: (i) SBP is superior in terms of accuracy whenever BP converges, and (ii) SBP obtains a unique, stable, and accurate solution whenever BP does not converge.

Self-Guided Belief Propagation -- A Homotopy Continuation Method

TL;DR

This work enhances BP and proposes self-guided belief propagation (SBP) that incorporates the pairwise potentials only gradually and provides a formal analysis to demonstrate that SBP finds the global optimum of the Bethe approximation for attractive models where all variables favor the same state.

Abstract

Belief propagation (BP) is a popular method for performing probabilistic inference on graphical models. In this work, we enhance BP and propose self-guided belief propagation (SBP) that incorporates the pairwise potentials only gradually. This homotopy continuation method converges to a unique solution and increases the accuracy without increasing the computational burden. We provide a formal analysis to demonstrate that SBP finds the global optimum of the Bethe approximation for attractive models where all variables favor the same state. Moreover, we apply SBP to various graphs with random potentials and empirically show that: (i) SBP is superior in terms of accuracy whenever BP converges, and (ii) SBP obtains a unique, stable, and accurate solution whenever BP does not converge.

Paper Structure

This paper contains 24 sections, 12 theorems, 34 equations, 7 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Probabilistic Graphical Models
  4. MODEL SPECIFICATION
  5. Frustrated, Balanced, and Unidirectional Models
  6. BELIEF PROPAGATION (BP)
  7. THE BETHE APPROXIMATION & RELATED WORK
  8. Solution Space of Binary Pairwise Models
  9. Combination of Fixed Points
  10. Self-guided belief propagation (SBP)
  11. Practical considerations
  12. Pseudocode
  13. Experiments
  14. Experimental settings
  15. Attractive models
  16. ...and 9 more sections

Key Result

Lemma 1

Let us consider a unidirectional model with loops, with $\theta_{i} = 0$, and with couplings strong enough to admit multiple (i.e., two) fixed points. Then, both fixed points $s$ and $t$ are symmetric, i.e., $m_{i}^{s}=-m_{i}^{t}$, and both fixed points have the same Bethe free energy, i.e., $\mathc

Figures (7)

  • Figure 1: Illustration of the model-classes specified in Section \ref{['subsec:model:classes']}: (a) frustrated; (b) balanced, and (c) its equivalent attractive model; and (d) unidirectional model. Solid lines depict attractive edges and dashed lines depict repulsive ones. The signs in the vertices are equal to the signs of the corresponding local fields $\theta_{i}$.
  • Figure 2: Evolution of the Bethe free energy $\mathcal{F}_B$ for Example \ref{['ex:vanishing']}. We consider a complete graph with $N=4$ variables with homogeneous potentials, attractive edges, and $\theta_{i}=0$. $\mathcal{F}_B$ is evaluated for $\tilde{P}(x_i) ={\tilde{P}_{}}$ for (a) $J_{}=0.1$, (b) $J_{}=0.55$, and (c) $J_{}=0.7$.
  • Figure 3: Evolution of the Bethe free energy $\mathcal{F}_B$ for Example \ref{['ex:unidirectional']}. We consider a complete graph with $N=4$ variables with homogeneous potentials, attractive edges, and $\theta_{i}>0$. $\mathcal{F}_B$ is evaluated for $\tilde{P}(x_i) ={\tilde{P}_{}}$ for (a) $J_{}=0.25$, (b) $J_{}=0.62$, and (c) $J_{}=0.75$.
  • Figure 4: Example \ref{['ex:solPath']}: SBP proceeds along the smooth solution path and obtains accurate marginals despite instability of the terminal fixed point.
  • Figure 5: $\text{MSE}$ and number of iterations for: $\text{SBP}_{all}$ (blue), $\text{BP}^\circ$ (orange), and $\text{BP}_{\text{D}}^\circ$ (green); $\theta_{i} \sim \mathcal{U}(-0.5, 0.5)$ and (a) $J_{ij} \sim \mathcal{U}(0,\beta)$ (attractive model); (b) $J_{ij} \sim \mathcal{U}(-\beta,\beta)$ (general model). In terms of accuracy, SBP is superior in all scenarios, while increasing the number of iterations only slightly.
  • ...and 2 more figures

Theorems & Definitions (18)

  • Lemma 1
  • Example 1
  • Lemma 2
  • Example 2
  • Conjecture 1: RSB Assumption
  • Example 3
  • Example 4
  • Theorem 3: Property \ref{['prop:properties']}.1
  • Theorem 4: Property \ref{['prop:properties']}.2
  • Lemma 5
  • ...and 8 more