Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

System Information Decomposition

Aobo Lyu, Bing Yuan, Ou Deng, Mingzhe Yang, Jiang Zhang

TL;DR

All the information atoms are symmetric, which means the disentanglement of unique, redundant, and synergistic information from any specific target variable, which positions SID as a promising framework with the potential to foster a deeper understanding of higher-order relationships within complex systems across disciplines.

Abstract

To characterize the complex higher-order interactions among variables within a system, this study introduces a novel framework, termed System Information Decomposition (SID), aimed at decomposing the information entropy of variables into information atoms based on their interrelations. Diverging from the established Partial Information Decomposition (PID) framework, which predominantly concentrates on the directional interactions stemming from an array of source variables to a single target variable, SID adopts a holistic approach, scrutinizing the interactions across all variables within the system. Specifically, we proved all the information atoms are symmetric, which means the disentanglement of unique, redundant, and synergistic information from any specific target variable. Hence, our proposed SID framework can capture the symmetric pairwise and higher-order relationships among variables. This advance positions SID as a promising framework with the potential to foster a deeper understanding of higher-order relationships within complex systems across disciplines.

System Information Decomposition

TL;DR

All the information atoms are symmetric, which means the disentanglement of unique, redundant, and synergistic information from any specific target variable, which positions SID as a promising framework with the potential to foster a deeper understanding of higher-order relationships within complex systems across disciplines.

Abstract

To characterize the complex higher-order interactions among variables within a system, this study introduces a novel framework, termed System Information Decomposition (SID), aimed at decomposing the information entropy of variables into information atoms based on their interrelations. Diverging from the established Partial Information Decomposition (PID) framework, which predominantly concentrates on the directional interactions stemming from an array of source variables to a single target variable, SID adopts a holistic approach, scrutinizing the interactions across all variables within the system. Specifically, we proved all the information atoms are symmetric, which means the disentanglement of unique, redundant, and synergistic information from any specific target variable. Hence, our proposed SID framework can capture the symmetric pairwise and higher-order relationships among variables. This advance positions SID as a promising framework with the potential to foster a deeper understanding of higher-order relationships within complex systems across disciplines.
Paper Structure (24 sections, 6 theorems, 42 equations, 8 figures, 1 table)

This paper contains 24 sections, 6 theorems, 42 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Information Decomposition
  3. Information Theory Framework
  4. Partial Information Decomposition Framework
  5. A Set-theoretic Understanding of PID
  6. System Information Decomposition
  7. Extension of PID in a System Scenario
  8. Properties of Information Atoms
  9. SID and Information Measure
  10. Case Studies
  11. Calculation of SID
  12. Properties of Calculation Methods for SID
  13. A Calculation Formula
  14. An Approximate Method by Neural Information Squeezer
  15. Discussion
  16. ...and 9 more sections

Key Result

Theorem 1

Let $X_{1},\cdots , X_{n}$ be the variables in a system. The redundant information is equal irrespective of the chosen target variable. Formally, we write $Red(X_{i}:{X_{1},\cdots , X_{n}}\setminus X_{i}) = Red(X_{j}:{X_{1},\cdots , X_{n}}\setminus X_{j}), \forall i, j \in \{1\cdots n\}$.

Figures (8)

  • Figure 1: Information Theory Venn Diagram.
  • Figure 2: Venn Diagram of PID.
  • Figure 3: Venn diagram from different perspectives of PID.
  • Figure 4: Venn diagram of SID's Preliminary version.
  • Figure 5: Venn diagram of SID's Formal Version.
  • ...and 3 more figures

Theorems & Definitions (17)

  • Definition 1: Set Intersection of Information kolchinsky2022novel
  • Definition 2: External Information
  • Theorem 1: Symmetry of Redundant Information
  • proof
  • Theorem 2: Symmetry of Unique Information
  • proof
  • Theorem 3: Symmetry of Synergistic Information
  • proof
  • Definition 3: System Information Decomposition Framework
  • Corollary 1: Joint Entropy Decomposition
  • ...and 7 more