Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Avoiding Non-Integrable Beliefs in Expectation Propagation

Zilu Zhao, Jichao Chen, Dirk Slock

Abstract

Expectation Propagation (EP) is a widely used iterative message-passing algorithm that decomposes a global inference problem into multiple local ones. It approximates marginal distributions as ``beliefs'' using intermediate functions called ``messages''. It has been shown that the stationary points of EP are the same as corresponding constrained Bethe Free Energy (BFE) optimization problem. Therefore, EP is an iterative method of optimizing the constrained BFE. However, the iterative method may fall out of the feasible set of the BFE optimization problem, i.e., the beliefs are not integrable. In most literature, the authors use various methods to keep all the messages integrable. In most Bayesian estimation problems, limiting the messages to be integrable shrinks the actual feasible set. Furthermore, in extreme cases where the factors are not integrable, making the message itself integrable is not enough to have integrable beliefs. In this paper, two EP frameworks are proposed to ensure that EP has integrable beliefs. Both of the methods allows non-integrable messages. We then investigate the signal recovery problem in Generalized Linear Model (GLM) using our proposed methods.

Avoiding Non-Integrable Beliefs in Expectation Propagation

Abstract

Expectation Propagation (EP) is a widely used iterative message-passing algorithm that decomposes a global inference problem into multiple local ones. It approximates marginal distributions as ``beliefs'' using intermediate functions called ``messages''. It has been shown that the stationary points of EP are the same as corresponding constrained Bethe Free Energy (BFE) optimization problem. Therefore, EP is an iterative method of optimizing the constrained BFE. However, the iterative method may fall out of the feasible set of the BFE optimization problem, i.e., the beliefs are not integrable. In most literature, the authors use various methods to keep all the messages integrable. In most Bayesian estimation problems, limiting the messages to be integrable shrinks the actual feasible set. Furthermore, in extreme cases where the factors are not integrable, making the message itself integrable is not enough to have integrable beliefs. In this paper, two EP frameworks are proposed to ensure that EP has integrable beliefs. Both of the methods allows non-integrable messages. We then investigate the signal recovery problem in Generalized Linear Model (GLM) using our proposed methods.

Paper Structure

This paper contains 19 sections, 4 theorems, 112 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Main Contributions
  3. Analytic Continuation Expectation Propagation (ACEP)
  4. Constrained KLD Optimization in Gaussian Projection
  5. EP with Checking (EPC)
  6. System Model for Generalized Linear Model
  7. Derivation for GLM
  8. $f_{y_n}\to z_n$
  9. $f_{x_k}\to x_k$
  10. $f_{\mathbf{z}}\to x_k$
  11. $f_{\mathbf{z}}\to z_n$
  12. $z\to f_{y}$
  13. $z\to f_{\mathbf{z}}$
  14. $x\to f_{x}$
  15. $x\to f_{\mathbf{z}}$
  16. ...and 4 more sections

Key Result

Proposition 1

In ACEP, with proper initialization (e.g. all positive message precision), if the update of $\mu^{f_{\mathbf{z}}\to z}_{n}$, $\mu^{f_{y}\to z}_{n}$, $\mu^{z\to f_{y}}_{n}$ and $\mu^{z\to f_{\mathbf{z}}}_{n}$ follows the following order: the update of $\mu^{f_{\mathbf{z}}\to z}_{n}$ and $\mu^{z\to f_{\mathbf{z}}}_{n}$ does not need to evaluate the threshold. $\blacktriangleleft$$\blacktriangleleft

Figures (1)

  • Figure 1: CDF of NMSE

Theorems & Definitions (10)

  • Proposition 1
  • proof
  • Remark
  • Proposition 2
  • proof
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Remark