Null models for comparing information decomposition across complex systems

TL;DR

This work introduces Null Models for Information Theory (NuMIT), a null model-based non-linear normalisation procedure which improves upon standard entropy-based normalisation approaches and overcomes their limitations.

Abstract

A key feature of information theory is its universality, as it can be applied to study a broad variety of complex systems. However, many information-theoretic measures can vary significantly even across systems with similar properties, making normalisation techniques essential for allowing meaningful comparisons across datasets. Inspired by the framework of Partial Information Decomposition (PID), here we introduce Null Models for Information Theory (NuMIT), a null model-based non-linear normalisation procedure which improves upon standard entropy-based normalisation approaches and overcomes their limitations. We provide practical implementations of the technique for systems with different statistics, and showcase the method on synthetic models and on human neuroimaging data. Our results demonstrate that NuMIT provides a robust and reliable tool to characterise complex systems of interest, allowing cross-dataset comparisons and providing a meaningful significance test for PID analyses.

Figures (14)

• Figure 1: Synergy and redundancy values for the bivariate Gaussian system given by Eqs. \ref{['eq:gauss_pid_distr']} and \ref{['eq:Gaussian_parameters']} for $g\in[1,100]$. Line styles represent raw atoms (dashed), atoms normalised by total mutual information (NMI, dotted), and atoms normalised by our proposed null-model procedure (NuMIT, solid).
• Figure 2: (a) Distributions of redundancy, unique information, and synergy with MMI definition for TMI=1.0 nat and (b) TMI=3.0 nat. (c) PID-atom averages for different values of mutual information over random Gaussian systems with 2 univariate sources and a univariate target. (d) Same as (c) but with NMI-normalised PID atoms.
• Figure 3: Quantiles of the PID atoms for Gaussian models of Eq. \ref{['eq:gauss_pid_distr_mult']} for various noise levels $g\in[1,100]$. (a) Predominantly redundant system (Eq. \ref{['eq:max_red_sys']}), (b) predominantly unique system (Eq. \ref{['eq:max_un_sys']}), (c) predominantly synergistic system (Eq. \ref{['eq:max_syn_sys']}).
• Figure 4: PID-atom distributions for all subjects under different drugs and placebo effects using MMI definition. From left to right, results refer to LSD, ketamine, and psilocybin drugs. Panel rows represent (a) the raw values of PID atoms, (b) the NMI-normalised PID atoms, and (c) the NuMIT-normalised PID atoms. The dashed black lines are drawn at zero. (P-values calculated with a one-sample t-test against the zero-mean null hypothesis. *: $p<0.05$; **: $p<0.01$; ***: $p<0.001$).
• Figure 5: Regression models for NMI- and NuMIT-normalised (a) synergies and (b) redundancies between $\text{CCS}$ and MMI PID definitions, for LSD, ketamine, and psilocybin drugs. $\Delta$ indicates the differences between drug and placebo in PID atoms obtained with either PID (MMI or CCS).