Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Higher-order Common Information

Jan Østergaard

TL;DR

This work defines higher-order common information $R_\ell$ to quantify what multiple arbitrarily distributed variables share, addressing the inadequacy of pairwise metrics like $R_2$ for $n>2$. It introduces sufficient common information (SCI) and a hierarchy of minimal SCI sets $\mathcal{T}_\ell$, proving lower bounds $R_\ell \ge \min_{\phi\in\mathcal{T}_{\ell-1}^u,\phi'\in\mathcal{T}_1} I(\phi;\phi')$ and deriving computable bounds for Gaussian and discrete sources. A practical estimation pipeline for general sources is proposed, combining mutual information estimators with SCI-based constructions (e.g., adding controlled Gaussian noise to form $T_{i,j}$) and extending to higher orders via iterative SCI construction. The method is validated on EEG data from a two-speech listening task, where third-order common information $R_3$ correlates with neural tracking of the attended envelope, offering insight beyond $R_2$ and suggesting utility for uncovering latent dependencies in time-series data.

Abstract

We present a new notion $R_\ell$ of higher-order common information, which quantifies the information that $\ell\geq 2$ arbitrarily distributed random variables have in common. We provide analytical lower bounds on $R_3$ and $R_4$ for jointly Gaussian distributed sources and provide computable lower bounds for $R_\ell$ for any $\ell$ and any sources. We also provide a practical method to estimate the lower bounds on, e.g., real-world time-series data. As an example, we consider EEG data acquired in a setup with competing acoustic stimuli. We demonstrate that $R_3$ has descriptive properties that is not in $R_2$. Moreover, we observe a linear relationship between the amount of common information $R_3$ communicated from the acoustic stimuli and to the brain and the corresponding cortical activity in terms of neural tracking of the envelopes of the stimuli.

Higher-order Common Information

TL;DR

This work defines higher-order common information to quantify what multiple arbitrarily distributed variables share, addressing the inadequacy of pairwise metrics like for . It introduces sufficient common information (SCI) and a hierarchy of minimal SCI sets , proving lower bounds and deriving computable bounds for Gaussian and discrete sources. A practical estimation pipeline for general sources is proposed, combining mutual information estimators with SCI-based constructions (e.g., adding controlled Gaussian noise to form ) and extending to higher orders via iterative SCI construction. The method is validated on EEG data from a two-speech listening task, where third-order common information correlates with neural tracking of the attended envelope, offering insight beyond and suggesting utility for uncovering latent dependencies in time-series data.

Abstract

We present a new notion of higher-order common information, which quantifies the information that arbitrarily distributed random variables have in common. We provide analytical lower bounds on and for jointly Gaussian distributed sources and provide computable lower bounds for for any and any sources. We also provide a practical method to estimate the lower bounds on, e.g., real-world time-series data. As an example, we consider EEG data acquired in a setup with competing acoustic stimuli. We demonstrate that has descriptive properties that is not in . Moreover, we observe a linear relationship between the amount of common information communicated from the acoustic stimuli and to the brain and the corresponding cortical activity in terms of neural tracking of the envelopes of the stimuli.
Paper Structure (17 sections, 6 theorems, 56 equations, 3 figures, 1 table)

This paper contains 17 sections, 6 theorems, 56 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Notation
  3. Paper organization
  4. Common information via conditional mutual information
  5. Higher-order common information via sufficient common information
  6. Common information --- Gaussian sources
  7. Common information --- discrete sources
  8. Estimating common information in general sources
  9. Approximations to the higher-order common information
  10. Second and third-order common information in EEG data
  11. Limitations
  12. Conclusions
  13. Proof of Lemma \ref{['lem:cmi']}
  14. Proof of Theorem \ref{['theo:mss']}
  15. Proof of Theorem \ref{['theo:gaussian3']}
  16. ...and 2 more sections

Key Result

Lemma 1

Let $X_i = X + N_i, i=1,\dotsc, n,$ where $X\sim \mathcal{N}(0,\sigma_X^2)$ and $N_i\sim \mathcal{N}(0,\sigma_N^2)$ are independent of each other for all $i$. Moreover, let $\rho \in [-1,1]$ be the pairwise correlation coefficient $\rho$ of $N_1,\dotsc, N_n,$ so that $\mathbb{E}[X_iX_j] = \rho\sigma $\square$

Figures (3)

  • Figure 1: Expression and bounds of Lemma \ref{['lem:cmi']} and Theorem \ref{['theo:gaussian3']}.
  • Figure 2: The pairwise ($R_2$) common information and lower bounds on the third- ($R_3^{\mathrm{lower}}$) and fourth-order ($R_4^{\mathrm{lower}}$) common information for four Gaussian variables: $X_i = X + N_i,\ \mathbb{E}[N_iN_j]=\rho \sigma_N^2$.
  • Figure 3: The top plot shows the probability density functions of the correlations values as a function of $\hat{R}_3$ for the target and distractor. The corresponding mean correlation ("distortion") values as a function of $\hat{R}_3$ for both the target and distractor stimuli are shown in the the bottom plot.

Theorems & Definitions (11)

  • Lemma 1
  • Definition 1
  • Definition 2: Sets of minimal sufficient common information
  • Remark 1
  • Definition 3: $\ell$-th order common information $R_\ell$
  • Theorem 1
  • Theorem 2
  • Corollary 1
  • Lemma 2
  • Theorem 3
  • ...and 1 more