Loopy Belief Propagation for Approximate Inference: An Empirical Study

TL;DR

This paper compares the marginals computed using loopy propagation to the exact ones in four Bayesian network architectures, including two real-world networks: ALARM and QMR, and finds that the loopy beliefs often converge and when they do, they give a good approximation to the correct marginals.

Abstract

Recently, researchers have demonstrated that loopy belief propagation - the use of Pearls polytree algorithm IN a Bayesian network WITH loops OF error- correcting codes.The most dramatic instance OF this IS the near Shannon - limit performance OF Turbo Codes codes whose decoding algorithm IS equivalent TO loopy belief propagation IN a chain - structured Bayesian network. IN this paper we ask : IS there something special about the error - correcting code context, OR does loopy propagation WORK AS an approximate inference schemeIN a more general setting? We compare the marginals computed using loopy propagation TO the exact ones IN four Bayesian network architectures, including two real - world networks : ALARM AND QMR.We find that the loopy beliefs often converge AND WHEN they do, they give a good approximation TO the correct marginals.However,ON the QMR network, the loopy beliefs oscillated AND had no obvious relationship TO the correct posteriors. We present SOME initial investigations INTO the cause OF these oscillations, AND show that SOME simple methods OF preventing them lead TO the wrong results.

Figures (9)

• Figure 1: The structure of the PYRAMID network. All nodes are binary and observations appear only on the bottom layer. Such networks occur often in image analysis where the bottom layer would correspond to pixels.
• Figure 2: The structure of a toyQMR network. This is a bipartite structure where the conditional distributions of the leaves are noisy-or's. The network shown represents one sample from randomly generated structures where the parents of each symptom were a random subset of the diseases.
• Figure 3: The structure of the ALARM network - a network constructed by medical experts for monitoring patients in intensive care.
• Figure 4: Correlation plots between the correct and approximate beliefs for the PYRAMID network, using (a) loopy propagation and (b) likelihood weighting with 200 samples.
• Figure 5: Correlation plots between the correct and approximate beliefs for the toyQMR network, using (a) loopy propagation and (b) likelihood weighting with 200 samples.