Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond the I-MMSE relation: derivatives of mutual information in Gaussian channels

Minh-Toan Nguyen

TL;DR

The paper addresses computing all higher-order derivatives of the mutual information with respect to the SNR in vector Gaussian channels, extending the classical I-MMSE relation to multi-dimensional settings. It introduces the combinatorial tau and bar-tau forms and derives a general all-orders derivative formula by combining the incremental-channel approach with the (non-rigorous) replica method, yielding a compact expression in terms of conditional expectations given the output Y. The main result states that, for a vector channel, the mixed partial derivatives of $I_{\mathbf{X}}(\boldsymbol{\lambda})$ equal conditional expectations of tau/ bar-tau evaluated at $\mathbf{Y}$, with a simple base case for first derivatives, and reveals a close relationship to the cumulant-moment formula. The work provides a conceptually unified framework that links high-order mutual information derivatives to a cumulant-like expansion and highlights directions for rigorous justification and potential applications in information-estimation theory.

Abstract

The I-MMSE formula connects two important quantities in information theory and estimation theory: the mutual information and the minimum mean-squared error (MMSE). It states that in a scalar Gaussian channel, the derivative of the mutual information with respect to the signal-to-noise ratio (SNR) is one-half of the MMSE. Although any derivative at a fixed order can be computed in principle, a general formula for all the derivatives is still unknown. In this paper, we derive this general formula for vector Gaussian channels. The obtained result is remarkably similar to the classic cumulant-moment relation in statistical theory.

Beyond the I-MMSE relation: derivatives of mutual information in Gaussian channels

TL;DR

The paper addresses computing all higher-order derivatives of the mutual information with respect to the SNR in vector Gaussian channels, extending the classical I-MMSE relation to multi-dimensional settings. It introduces the combinatorial tau and bar-tau forms and derives a general all-orders derivative formula by combining the incremental-channel approach with the (non-rigorous) replica method, yielding a compact expression in terms of conditional expectations given the output Y. The main result states that, for a vector channel, the mixed partial derivatives of equal conditional expectations of tau/ bar-tau evaluated at , with a simple base case for first derivatives, and reveals a close relationship to the cumulant-moment formula. The work provides a conceptually unified framework that links high-order mutual information derivatives to a cumulant-like expansion and highlights directions for rigorous justification and potential applications in information-estimation theory.

Abstract

The I-MMSE formula connects two important quantities in information theory and estimation theory: the mutual information and the minimum mean-squared error (MMSE). It states that in a scalar Gaussian channel, the derivative of the mutual information with respect to the signal-to-noise ratio (SNR) is one-half of the MMSE. Although any derivative at a fixed order can be computed in principle, a general formula for all the derivatives is still unknown. In this paper, we derive this general formula for vector Gaussian channels. The obtained result is remarkably similar to the classic cumulant-moment relation in statistical theory.
Paper Structure (9 sections, 5 theorems, 70 equations, 3 figures, 1 table)

This paper contains 9 sections, 5 theorems, 70 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Statement of results
  3. Tools
  4. A nice property of Gaussian channels
  5. Replica method
  6. A quick introduction to the replica method
  7. Mutual information and replicas
  8. Proofs
  9. Conclusion

Key Result

Proposition 1

$$ a) $\tau$ and $\bar{\tau}$ are multiquadratic. b) $\tau((X_i)_{i \in [n]}) = 0$ if $[n]$ can be divided into two disjoint, non-empty sets $I$ and $J$ such that $(X_i)_{i \in I}$ and $(X_j)_{j \in J}$ are independent. c) The forms $\tau$ and $\bar{\tau}$ are related by the following identity

Figures (3)

  • Figure 1: Loop-free graphs with four edges.
  • Figure 2: Different labelings of a graph.
  • Figure 3: Example of two equivalent labelings, describing the same partition $(\mathopen{}\mathclose{\left\{ 3,4 \right\}, \mathopen{}\mathclose{\left\{ 1,2,3} \right\}, \mathopen{}\mathclose{\left\{ 1,2,4} \right\})$

Theorems & Definitions (20)

  • Definition 1
  • Definition 2
  • Example 1
  • Proposition 1
  • Theorem 1
  • Remark 1
  • Remark 2
  • Example 2
  • Example 3
  • Example 4
  • ...and 10 more