Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Information Geometry Interpretation for Approximate Message Passing

Bingyan Liu, An-An Lu, Mingrui Fan, Jiyuan Yang, Xiqi Gao

TL;DR

It is proved that the AIGA is equivalent to the approximate message passing (AMP) algorithm, and intrinsic results offer a new perspective for the AMP algorithm, and clues for understanding and improving stochastic reasoning methods.

Abstract

In this paper, we propose an information geometry (IG) framework to solve the standard linear regression problem. The proposed framework is an extension of the one for computing the mean of complex multivariate Gaussian distribution. By applying the proposed framework, the information geometry approach (IGA) and the approximate information geometry approach (AIGA) for basis pursuit de-noising (BPDN) in standard linear regression are derived. The framework can also be applied to other standard linear regression problems. With the transformations of natural and expectation parameters of Gaussian distributions, we then show the relationship between the IGA and the message passing (MP) algorithm. Finally, we prove that the AIGA is equivalent to the approximate message passing (AMP) algorithm. These intrinsic results offer a new perspective for the AMP algorithm, and clues for understanding and improving stochastic reasoning methods.

An Information Geometry Interpretation for Approximate Message Passing

TL;DR

It is proved that the AIGA is equivalent to the approximate message passing (AMP) algorithm, and intrinsic results offer a new perspective for the AMP algorithm, and clues for understanding and improving stochastic reasoning methods.

Abstract

In this paper, we propose an information geometry (IG) framework to solve the standard linear regression problem. The proposed framework is an extension of the one for computing the mean of complex multivariate Gaussian distribution. By applying the proposed framework, the information geometry approach (IGA) and the approximate information geometry approach (AIGA) for basis pursuit de-noising (BPDN) in standard linear regression are derived. The framework can also be applied to other standard linear regression problems. With the transformations of natural and expectation parameters of Gaussian distributions, we then show the relationship between the IGA and the message passing (MP) algorithm. Finally, we prove that the AIGA is equivalent to the approximate message passing (AMP) algorithm. These intrinsic results offer a new perspective for the AMP algorithm, and clues for understanding and improving stochastic reasoning methods.
Paper Structure (22 sections, 2 theorems, 8 equations, 5 figures, 2 tables, 2 algorithms)

This paper contains 22 sections, 2 theorems, 8 equations, 5 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation and Information Geometry Framework
  3. Problem Formulation
  4. Information Geometry Framework
  5. Auxiliary manifolds and the $e$-condition
  6. The $m$-projection and the $m$-condition
  7. Information Geometry Approach for BPDN
  8. Auxiliary and Target PDFs
  9. Relation between the Extra PDF and the Target PDF
  10. Algorithm Derivation
  11. Approximate Information Geometry Approach for BPDN
  12. Approximations of Beliefs
  13. Updating Process with Approximations
  14. The equivalence of AMP and AIGA
  15. Review of MP and IGA
  16. ...and 7 more sections

Key Result

Theorem 1

Define the beliefs $\bm\xi_m$, $\bm\Xi_m$ as $\bm\xi_m=\bm\lambda_m^0 - \bm\lambda_m$ and $\bm\Xi_m = \bm\Lambda_m^0 -\bm\Lambda_m$. Then, the beliefs are given by where $r_m=1+\mathbf a_m^T{\bm\Lambda}_m^{-1}\mathbf a_m$ and $\mathbf L_m = \mathbf I\odot\mathbf a_m\mathbf a_m^T$.

Figures (5)

  • Figure 1: $e$-condition
  • Figure 2: $m$-condition
  • Figure 3: Factor graph of \ref{['eq:postupost']}
  • Figure 4: Convergence performance of IGA, AMP and AIGA, where $M=512$, $N=1024$, $\rho = 0.05$, $\text{SNR}=\{20,40\}\text{dB}$ and $\kappa=0.05$.
  • Figure 5: Convergence performance of IGA, AMP and AIGA, where $M=256$, $N=512$, $\rho = 0.05$, $\text{SNR}=\{20,40\}\text{dB}$ and $\kappa=0.05$.

Theorems & Definitions (4)

  • Theorem 1
  • proof
  • Theorem 2
  • proof