Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nonnegative Decomposition of Multivariate Information

Paul L. Williams, Randall D. Beer

TL;DR

This paper proposes a principled, nonnegative decomposition of multivariate information by redefining redundancy as I_min and organizing information contributions into a redundancy lattice. It derives a partial information decomposition (PI) with nonnegative atoms that correspond to unique, redundant, and synergistic contributions from source subsets, resolving interpretive issues with interaction information. Through a Mobius-inversion-based PI-function, the framework yields a clear, interpretable breakdown of I(S;R) and explains when and why interaction information can be negative. The approach offers a scalable conceptual and mathematical tool for understanding structure in multivariate systems, with applications spanning neuroscience, genetics, physics, and machine learning, while highlighting computational challenges for larger variable sets.

Abstract

Of the various attempts to generalize information theory to multiple variables, the most widely utilized, interaction information, suffers from the problem that it is sometimes negative. Here we reconsider from first principles the general structure of the information that a set of sources provides about a given variable. We begin with a new definition of redundancy as the minimum information that any source provides about each possible outcome of the variable, averaged over all possible outcomes. We then show how this measure of redundancy induces a lattice over sets of sources that clarifies the general structure of multivariate information. Finally, we use this redundancy lattice to propose a definition of partial information atoms that exhaustively decompose the Shannon information in a multivariate system in terms of the redundancy between synergies of subsets of the sources. Unlike interaction information, the atoms of our partial information decomposition are never negative and always support a clear interpretation as informational quantities. Our analysis also demonstrates how the negativity of interaction information can be explained by its confounding of redundancy and synergy.

Nonnegative Decomposition of Multivariate Information

TL;DR

This paper proposes a principled, nonnegative decomposition of multivariate information by redefining redundancy as I_min and organizing information contributions into a redundancy lattice. It derives a partial information decomposition (PI) with nonnegative atoms that correspond to unique, redundant, and synergistic contributions from source subsets, resolving interpretive issues with interaction information. Through a Mobius-inversion-based PI-function, the framework yields a clear, interpretable breakdown of I(S;R) and explains when and why interaction information can be negative. The approach offers a scalable conceptual and mathematical tool for understanding structure in multivariate systems, with applications spanning neuroscience, genetics, physics, and machine learning, while highlighting computational challenges for larger variable sets.

Abstract

Of the various attempts to generalize information theory to multiple variables, the most widely utilized, interaction information, suffers from the problem that it is sometimes negative. Here we reconsider from first principles the general structure of the information that a set of sources provides about a given variable. We begin with a new definition of redundancy as the minimum information that any source provides about each possible outcome of the variable, averaged over all possible outcomes. We then show how this measure of redundancy induces a lattice over sets of sources that clarifies the general structure of multivariate information. Finally, we use this redundancy lattice to propose a definition of partial information atoms that exhaustively decompose the Shannon information in a multivariate system in terms of the redundancy between synergies of subsets of the sources. Unlike interaction information, the atoms of our partial information decomposition are never negative and always support a clear interpretation as informational quantities. Our analysis also demonstrates how the negativity of interaction information can be explained by its confounding of redundancy and synergy.

Paper Structure

This paper contains 11 sections, 7 theorems, 29 equations, 9 figures.

Table of Contents

  1. Introduction
  2. The Structure of Multivariate Information
  3. Measuring Redundancy
  4. Partial Information Decomposition
  5. Why Interaction Information is Sometimes Negative
  6. Discussion
  7. Measures of Specific Information
  8. Lattice Theory Definitions
  9. $\mathcal{A}({\bf R})$ and the Redundancy Lattice
  10. Supporting Proofs
  11. Supplementary Figures

Key Result

Theorem 1

$I(S=s; {\bf A})$ is nonnegative.

Figures (9)

  • Figure 1: Structure of multivariate information for 3 variables. Labelled regions correspond to unique information (Unq), redundancy (Rdn), and synergy (Syn).
  • Figure 2: Redundancy lattice for (A) 3 and (B) 4 variables.
  • Figure 3: Partial information diagrams for (A) 3 and (B) 4 variables.
  • Figure 4: Probability distributions for $S \in \{0,1,2\}$ and $R_1, R_2 \in \{0,1\}$. Black tiles represent equiprobable outcomes. White tiles are zero-probability outcomes.
  • Figure 5: PI-decomposition of interaction information for (A) 3 and (B) 4 variables. Blue and red regions represent PI-terms that are added and subtracted, respectively. The green region in (B) represents a PI-term that is subtracted twice.
  • ...and 4 more figures

Theorems & Definitions (23)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Definition 5
  • Definition 6
  • Definition 7
  • Definition 8
  • Definition 9
  • Theorem 1
  • ...and 13 more