Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Information-theoretic limits and approximate message-passing for high-dimensional time series

Daria Tieplova, Samriddha Lahiry, Jean Barbier

TL;DR

This work analyzes high-dimensional time series where the number of features grows in proportion to samples, without sparsity, by deriving a single-letter information-theoretic description of the stochastic regression model with a diagonal AR(1) driver. It presents a replica-symmetric variational formula for the normalized mutual information and the limiting MMSE, and extends to general diagonal eigenvalue structures via a block-approximation approach. The authors provide a rigorous adaptive interpolation proof of the replica formula and connect MMSE to the measurement MMSE through I-MMSE relations, with explicit expressions involving scalar-channel mutual informations and KMS-like spectral integrals. Empirically, VAMP matches theoretical predictions in right-rotationally invariant settings and shows robust performance even with block-structured, non-rotationally invariant designs, highlighting practical viability for inference in high-dimensional time-series settings.

Abstract

High-dimensional time series appear in many scientific setups, demanding a nuanced approach to model and analyze the underlying dependence structure. Theoretical advancements so far often rely on stringent assumptions regarding the sparsity of the underlying signal. In non-sparse regimes, analyses have primarily focused on linear regression models with the design matrix having independent rows. In this paper, we expand the scope by investigating a high-dimensional time series model wherein the number of features grows proportionally to the number of sampling points, without assuming sparsity in the signal. Specifically, we consider the stochastic regression model and derive a single-letter formula for the normalized mutual information between observations and the signal, as well as for minimum mean-square errors. We also empirically study the vector approximate message passing VAMP algorithm and show that, despite the lack of theoretical guarantees, its performance for inference in our time series model is robust and often statistically optimal.

Information-theoretic limits and approximate message-passing for high-dimensional time series

TL;DR

This work analyzes high-dimensional time series where the number of features grows in proportion to samples, without sparsity, by deriving a single-letter information-theoretic description of the stochastic regression model with a diagonal AR(1) driver. It presents a replica-symmetric variational formula for the normalized mutual information and the limiting MMSE, and extends to general diagonal eigenvalue structures via a block-approximation approach. The authors provide a rigorous adaptive interpolation proof of the replica formula and connect MMSE to the measurement MMSE through I-MMSE relations, with explicit expressions involving scalar-channel mutual informations and KMS-like spectral integrals. Empirically, VAMP matches theoretical predictions in right-rotationally invariant settings and shows robust performance even with block-structured, non-rotationally invariant designs, highlighting practical viability for inference in high-dimensional time-series settings.

Abstract

High-dimensional time series appear in many scientific setups, demanding a nuanced approach to model and analyze the underlying dependence structure. Theoretical advancements so far often rely on stringent assumptions regarding the sparsity of the underlying signal. In non-sparse regimes, analyses have primarily focused on linear regression models with the design matrix having independent rows. In this paper, we expand the scope by investigating a high-dimensional time series model wherein the number of features grows proportionally to the number of sampling points, without assuming sparsity in the signal. Specifically, we consider the stochastic regression model and derive a single-letter formula for the normalized mutual information between observations and the signal, as well as for minimum mean-square errors. We also empirically study the vector approximate message passing VAMP algorithm and show that, despite the lack of theoretical guarantees, its performance for inference in our time series model is robust and often statistically optimal.
Paper Structure (25 sections, 17 theorems, 205 equations, 3 figures)

This paper contains 25 sections, 17 theorems, 205 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Setting
  3. Related literature
  4. Notations and outline of the paper
  5. Main results
  6. Mutual information
  7. General $\mathbf{A}_p$
  8. Special Cases
  9. Minimum mean-square errors
  10. Numerical experiments with VAMP
  11. Proofs
  12. Proof of the replica formula by the adaptive interpolation method
  13. Step 1
  14. Step 2
  15. Step 3
  16. ...and 10 more sections

Key Result

Theorem 2.1

Assume that the signal $\boldsymbol{\beta}$ has i.i.d. entries with prior $P_0$ with compact support and matrix $\mathbf{A}_p$ is diagonal with a number of different eigenvalues $(\lambda_1,\ldots,\lambda_k)$ independent of $N,p$ (see Case 3 or diagonal_lambda_model). Then we have

Figures (3)

  • Figure 1: On the left MMSE versus $N/p$ for Gaussian prior. We take $p=2100$, and $\mathbf{A}_p=0$ (red) and $\mathbf{A}_p={\rm diag}(0.9,0.7,0.5,0.3,0.1)$ (blue). On the right we see models with $\boldsymbol{\beta}$ drawn from Rademacher prior with $p=2100$, $\sigma^2=0.1$. Blue line represents theoretically obtained MMSE for the case with i.i.d. Gaussian design matrix ($\mathbf{A}_p=0$) and on the red - the case of right rotation invariant design matrix ($\mathbf{A}_p=0.9\mathbf{I}_p$). Bars represents the span of MSE obtained through the VAMP algorithm for 50 instances.
  • Figure 2: On the right MMSE versus $c_N=N/p$ for models with Rademacher prior, $\sigma^2=0.1$, $p=2100$, and two different choices of $\mathbf{A}_p$. Continuous lines represent the theoretically obtained MMSE, \ref{['eq:mmse_conj']}, while the bars show the span of MSE obtained from the VAMP algorithm for 50 instances of the problem. On the left we see MMSE given by \ref{['eq:mmse_conj']} versus $1/\sigma^2$ (blue), and MSE of VAMP averaged over 50 instances with signal with entries drawn from the Rademacher prior with $p=2100$ and parameters $N/p=0.3$, with $\mathbf{A}_p=\{0.9,0.1\}$.
  • Figure 3: YMMSE versus $c_N=N/p$ for models with Rademacher prior, $\sigma^2=0.1$, $p=2100$, and different choices of $\mathbf{A}_p$. Continuous lines represent the theoretically obtained YMMSE, while "+" show the YMMSE averaged over 50 instances obtained from the VAMP algorithm. The plot on the right shows two models with right rotationally invariant design matrices ($\mathbf{A}_p=0$ and $\mathbf{A}_p=0.9 \mathbf{I}_p$ and on the left we see two models with block right rotationally invariant design matrices ($\mathbf{A}_p=\{0.9,0.1\}$ and $\mathbf{A}_p=\{0.9,0.8,0.7\}$).

Theorems & Definitions (29)

  • Theorem 2.1: Replica formula for the stochastic regression model
  • Remark 2.1
  • Theorem 2.2: Replica formula with general eigenvalue distribution
  • Theorem 2.3: Measurement MMSE, and connection to the block MMSEs
  • Lemma 4.1: Time derivative
  • proof
  • Lemma 4.2: Overlap concentration
  • proof
  • Lemma 4.3: Fundamental identity
  • Lemma 4.4: Upper bound
  • ...and 19 more