Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Logarithmic Decomposition and a Signed Measure Space for Entropy

Keenan J. A. Down, Pedro A. M. Mediano

TL;DR

This work introduces a logarithmic decomposition (LD) that refines Yeung's I-measure by constructing a maximal outcome-space ΔΩ and an additive signed measure μ (interior loss) via Möbius inversion of the total entropy loss. The LD framework yields a fine-grained, atom-wise representation of information where the entropy of a variable equals the measure of its logarithmic content, H(X)=μ(ΔX), and mutual information is μ(ΔX∩ΔY), thereby providing a natural refinement of the I-measure. It further captures quantities like Gács–Körner common information, the minimum sufficient statistic, and functional common information, and extends to multiplicity-based metrics (TC, DTC, O-information) by counting atoms with multiplicity; the framework also accommodates continuous distributions through a refined limit construction. Importantly, LD distinguishes the Dyadic and Triadic systems without invoking PID, suggesting new avenues for analyzing information structure and challenging the necessity of some PID extensions. The approach offers potential benefits for information processing in neuroscience and explainable AI, and lays groundwork for deeper algebraic studies of information decomposition.

Abstract

The Shannon entropy of a random variable has much behaviour analogous to a signed measure. Previous work has explored this connection by defining a signed measure on abstract sets, which are taken to represent the information that different random variables contain. This construction is sufficient to derive many measure-theoretical counterparts to information quantities such as the mutual information (the intersection of sets), the joint entropy (the union of sets), and the conditional entropy (the difference of sets). Here we provide concrete characterisations of these abstract sets and a corresponding signed measure by extending the approach used by Yeung to all possible outcomes in an outcome space $Ω$, and in doing so we demonstrate that there exists a much finer decomposition with intuitive properties which we call the logarithmic decomposition (LD). We show that this signed measure space has the useful property that its logarithmic atoms are easily characterised with negative or positive entropy, depending only on their structure, while also being consistent with Yeung's I-measure. We present the usability of our approach by re-examining the Gács-Körner common information and minimally sufficient statistics from this new geometric perspective and characterising it in terms of our logarithmic atoms -- a property we call logarithmic decomposability. We present possible extensions of this construction to continuous probability distributions before discussing implications for quality-led information theory. As a motivating example, we apply our new decomposition to the Dyadic and Triadic systems of James and Crutchfield and show that, in contrast to the I-measure alone, our decomposition is able to qualitatively distinguish between them.

A Logarithmic Decomposition and a Signed Measure Space for Entropy

TL;DR

This work introduces a logarithmic decomposition (LD) that refines Yeung's I-measure by constructing a maximal outcome-space ΔΩ and an additive signed measure μ (interior loss) via Möbius inversion of the total entropy loss. The LD framework yields a fine-grained, atom-wise representation of information where the entropy of a variable equals the measure of its logarithmic content, H(X)=μ(ΔX), and mutual information is μ(ΔX∩ΔY), thereby providing a natural refinement of the I-measure. It further captures quantities like Gács–Körner common information, the minimum sufficient statistic, and functional common information, and extends to multiplicity-based metrics (TC, DTC, O-information) by counting atoms with multiplicity; the framework also accommodates continuous distributions through a refined limit construction. Importantly, LD distinguishes the Dyadic and Triadic systems without invoking PID, suggesting new avenues for analyzing information structure and challenging the necessity of some PID extensions. The approach offers potential benefits for information processing in neuroscience and explainable AI, and lays groundwork for deeper algebraic studies of information decomposition.

Abstract

The Shannon entropy of a random variable has much behaviour analogous to a signed measure. Previous work has explored this connection by defining a signed measure on abstract sets, which are taken to represent the information that different random variables contain. This construction is sufficient to derive many measure-theoretical counterparts to information quantities such as the mutual information (the intersection of sets), the joint entropy (the union of sets), and the conditional entropy (the difference of sets). Here we provide concrete characterisations of these abstract sets and a corresponding signed measure by extending the approach used by Yeung to all possible outcomes in an outcome space , and in doing so we demonstrate that there exists a much finer decomposition with intuitive properties which we call the logarithmic decomposition (LD). We show that this signed measure space has the useful property that its logarithmic atoms are easily characterised with negative or positive entropy, depending only on their structure, while also being consistent with Yeung's I-measure. We present the usability of our approach by re-examining the Gács-Körner common information and minimally sufficient statistics from this new geometric perspective and characterising it in terms of our logarithmic atoms -- a property we call logarithmic decomposability. We present possible extensions of this construction to continuous probability distributions before discussing implications for quality-led information theory. As a motivating example, we apply our new decomposition to the Dyadic and Triadic systems of James and Crutchfield and show that, in contrast to the I-measure alone, our decomposition is able to qualitatively distinguish between them.
Paper Structure (30 sections, 23 theorems, 122 equations, 11 figures)

This paper contains 30 sections, 23 theorems, 122 equations, 11 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Relation to Partial Information Decomposition (PID) and other decomposition methods
  4. Main contributions
  5. An explicit definition for abstract information spaces
  6. Construction of a signed measure
  7. Properties of entropy loss, $L$
  8. Properties of the measure $\mu$
  9. Uniqueness of the Measure
  10. Quantities of Information
  11. Mutual, Conditional and Co-information
  12. Gács-Körner Common Information
  13. Functional common information
  14. The minimum sufficient statistic
  15. Quantities with information multiplicity
  16. ...and 15 more sections

Key Result

Theorem 11

Let $\Omega$ be a finite set of outcomes and let $\Sigma$ be the $\sigma$-algebra generated by all of the elements $b\in \Delta\Omega$. For $S\subseteq \Delta\Omega$ define $\mu(S) = \sum_{b\in S} \mu(b)$. Then $(\Delta\Omega, \Sigma, \mu)$ is a finite signed measure space.

Figures (11)

  • Figure 1: The highlighted triangle along with its boundary corresponds to the subset $\{b_{12}, b_{14}, b_{24}, b_{124} \}$.
  • Figure 2: Treating $\Omega$ itself as a random variable (possibly as the joint of all variables being considered, $\Omega$ has entropy $H(\Omega) = 1.95$ bits. After merging the three outcomes, $X$ has entropy $H(X) = 0.84$ bits, having lost more than a bit of entropy. There are different intermediate ways of merging outcomes which lose less, but different entropies en route to $X$.
  • Figure 3: The co-information between three variables.
  • Figure 4: Two random variables on the set $\Omega = \{1, 2, 3, 4\}$ with some illustrative probabilities.
  • Figure 5: Logarithmic atoms of $\Omega = \{1,2, 3, 4\}$ in the space $\Delta \Omega$.
  • ...and 6 more figures

Theorems & Definitions (86)

  • Definition 1
  • Example 2
  • Remark 3
  • Remark 4
  • Definition 5
  • Remark 6
  • Remark 7
  • Definition 8
  • Example 9
  • Remark 10
  • ...and 76 more