Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Expectations in Expectation Propagation

Zilu Zhao, Fangqing Xiao, Dirk Slock

TL;DR

The paper investigates expectation-propagation in linear models, focusing on the challenge of messages with infinite integrals (negative variances) that can block EP progress. It analyzes reVAMP, proving that likelihood beliefs are finite when priors are finite, and proposes non-persistent and persistent blocking-prevention strategies along with Analytic Continuation reVAMP (ACReVAMP) to avoid negative variances. Through theory and simulations on sparse and BPSK problems, it shows that allowing negative-variance messages can improve continuous-valued recovery, while suppressing them helps discrete-like signals, offering practical strategies for robust EP in linear inference. The work clarifies the relationship between EP messages and beliefs and delivers computationally efficient updates via low-rank adjustments and analytic continuation.

Abstract

Expectation Propagation (EP) is a widely used message-passing algorithm that decomposes a global inference problem into multiple local ones. It approximates marginal distributions (beliefs) using intermediate functions (messages). While beliefs must be proper probability distributions that integrate to one, messages may have infinite integral values. In Gaussian-projected EP, such messages take a Gaussian form and appear as if they have "negative" variances. Although allowed within the EP framework, these negative-variance messages can impede algorithmic progress. In this paper, we investigate EP in linear models and analyze the relationship between the corresponding beliefs. Based on the analysis, we propose both non-persistent and persistent approaches that prevent the algorithm from being blocked by messages with infinite integral values. Furthermore, by examining the relationship between the EP messages in linear models, we develop an additional approach that avoids the occurrence of messages with infinite integral values.

Expectations in Expectation Propagation

TL;DR

The paper investigates expectation-propagation in linear models, focusing on the challenge of messages with infinite integrals (negative variances) that can block EP progress. It analyzes reVAMP, proving that likelihood beliefs are finite when priors are finite, and proposes non-persistent and persistent blocking-prevention strategies along with Analytic Continuation reVAMP (ACReVAMP) to avoid negative variances. Through theory and simulations on sparse and BPSK problems, it shows that allowing negative-variance messages can improve continuous-valued recovery, while suppressing them helps discrete-like signals, offering practical strategies for robust EP in linear inference. The work clarifies the relationship between EP messages and beliefs and delivers computationally efficient updates via low-rank adjustments and analytic continuation.

Abstract

Expectation Propagation (EP) is a widely used message-passing algorithm that decomposes a global inference problem into multiple local ones. It approximates marginal distributions (beliefs) using intermediate functions (messages). While beliefs must be proper probability distributions that integrate to one, messages may have infinite integral values. In Gaussian-projected EP, such messages take a Gaussian form and appear as if they have "negative" variances. Although allowed within the EP framework, these negative-variance messages can impede algorithmic progress. In this paper, we investigate EP in linear models and analyze the relationship between the corresponding beliefs. Based on the analysis, we propose both non-persistent and persistent approaches that prevent the algorithm from being blocked by messages with infinite integral values. Furthermore, by examining the relationship between the EP messages in linear models, we develop an additional approach that avoids the occurrence of messages with infinite integral values.

Paper Structure

This paper contains 32 sections, 3 theorems, 62 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Main Contribution
  3. Notations
  4. Expectation Propagation
  5. Problem Formulation
  6. Useful Relations
  7. Matrix Inversion Lemma
  8. Matrix Determinant Lemma
  9. Multivariate Gaussian Reproduction Lemma
  10. System Model
  11. Brief Introduction to reVAMP
  12. Ideal reVAMP: Message from $f_{{\bm y}}$ to ${\bm x}$
  13. Ideal reVAMP: Message from $f_{x_{n(t)}}$ to $x_{n(t)}$
  14. Low-Rank Update
  15. Keeping Negative Variance
  16. ...and 17 more sections

Key Result

Lemma 1

If, in reVAMP, sequential updates are used and $\tau_{p_{n(t)}}(t+1)$ is updated via eq:asilomar45, then

Figures (3)

  • Figure 1: Sparse Signal Recovery
  • Figure 2: Sparse Signal Recovery
  • Figure 3: BPSK Signal Recovery

Theorems & Definitions (8)

  • Remark
  • Lemma 1
  • proof
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Remark