Exact Fractional Inference via Re-Parametrization & Interpolation between Tree-Re-Weighted- and Belief Propagation- Algorithms

TL;DR

Theoretical analysis is complemented by extensive experiments with models from Ising ensembles over planar and random graphs of medium and large sizes, which yields a number of interesting observations.

Abstract

Computing the partition function, $Z$, of an Ising model over a graph of $N$ \enquote{spins} is most likely exponential in $N$. Efficient variational methods, such as Belief Propagation (BP) and Tree Re-Weighted (TRW) algorithms, compute $Z$ approximately by minimizing the respective (BP- or TRW-) free energy. We generalize the variational scheme by building a $λ$-fractional interpolation, $Z^{(λ)}$, where $λ=0$ and $λ=1$ correspond to TRW- and BP-approximations, respectively. This fractional scheme -- coined Fractional Belief Propagation (FBP) -- guarantees that in the attractive (ferromagnetic) case $Z^{(TRW)} \geq Z^{(λ)} \geq Z^{(BP)}$, and there exists a unique (\enquote{exact}) $λ_*$ such that $Z=Z^{(λ_*)}$. Generalizing the re-parametrization approach of \citep{wainwright_tree-based_2002} and the loop series approach of \citep{chertkov_loop_2006}, we show how to express $Z$ as a product, $\forall λ:\ Z=Z^{(λ)}{\tilde Z}^{(λ)}$, where the multiplicative correction, ${\tilde Z}^{(λ)}$, is an expectation over a node-independent probability distribution built from node-wise fractional marginals. Our theoretical analysis is complemented by extensive experiments with models from Ising ensembles over planar and random graphs of medium and large sizes. Our empirical study yields a number of interesting observations, such as the ability to estimate ${\tilde Z}^{(λ)}$ with $O(N^{2::4})$ fractional samples and suppression of variation in $λ_*$ estimates with an increase in $N$ for instances from a particular random Ising ensemble, where $[2::4]$ indicates a range from $2$ to $4$. We also discuss the applicability of this approach to the problem of image de-noising.

Figures (6)

• Figure 1: The case of the attractive Ising Model (a) with non-zero field and random interaction, $\bm{h}, \bm{J} \sim \mathcal{U}(0,1)$ on a $3\times3$ planar grid; and (b) with zero field and random interaction, $\bm{J} \sim \mathcal{U}(0,1)$ on a $3\times3$ planar grid; (c) with non-zero field and random interaction, $\bm{h}, \bm{J} \sim \mathcal{U}(0,1)$ on a $K_9$ complete graph and (d) with zero field and random interaction, $\bm{J} \sim \mathcal{U}(0,1)$ on a $K_9$ complete graph. We show fractional log-partition function (minus fractional free energy) on the left and the respective correction factor ${\tilde{Z}}^{(\lambda)}$ on the right vs the fractional parameter, $\lambda$. We observe monotonicity and concavity of $\bar{F}^{(\lambda)}$ on $\lambda$.
• Figure 2: Planar zero-field Ising models for $n\times n$ grid with $J \sim \mathcal{U}(0,1)$. For each $n$, four different instances are generated by sampling uniformly at random from the unit interval and the exact values, $\lambda_*$, are shown by open symbols on each graph. (a) $10\times 10$ (b) $20\times 20$ (c) $30\times 30$ (d) $40\times 40$.
• Figure 3: Dependence of the sample-based estimate of $Z$ on the number of samples in the case of attractive Ising model for two different values of $\lambda$ in the case of (a) $4\times 4$, and (b) $5\times 5$ grids, where elements of ${\bm J}$ and ${\bm h}$ are drawn i.i.d. from $\mathcal{U}[0,1]$. The number of samples are $N^i$, where $i=1,2,3,4$ and $N = |\mathcal{V}|$ (that is $N=16$ in (a) and $N=25$ in (b).
• Figure 4: Different algorithms and their corresponding errors (listed below each image) for image de-noising.
• Figure 5: $F^{(\lambda)}$ vs $\lambda$ for a number of instances (shown in different colors) drawn for the Ising model ensembles over, (a) $3\times 3$ grid, and (b) $K_9$ graph, where elements of ${\bm J}$ and ${\bm h}$ are i.i.d. from $\mathcal{U}(0,1)$. Circles mark respective exact values, $\lambda_*$.

Theorems & Definitions (9)

• Theorem 3.1
• proof
• Remark
• Theorem 3.2
• proof
• Lemma 3.3
• proof
• Theorem 4.1
• proof