Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Shannon Information and Kolmogorov Complexity

Peter Grunwald, Paul Vitanyi

TL;DR

The paper juxtaposes Shannon entropy $H(X)$ with Kolmogorov complexity $K(x)$ to clarify their distinct aims: ensemble-based coding versus individual-object description. It shows that the expected Kolmogorov complexity under a computable distribution approximates Shannon entropy, and that probabilistic mutual information corresponds closely to its algorithmic analogue on average. By introducing the algorithmic structure function and algorithmic sufficiency, the authors connect MDL and rate-distortion theory to a purely descriptional framework, enabling model selection that balances meaningful structure with parsimony. The work also establishes that probabilistic and algorithmic notions of information obey analogous inequalities, with algorithmic quantities matching probabilistic ones up to additive (often logarithmic) terms, and it highlights practical pathways through universal coding and MDL as bridges between theory and data analysis.

Abstract

We compare the elementary theories of Shannon information and Kolmogorov complexity, the extent to which they have a common purpose, and where they are fundamentally different. We discuss and relate the basic notions of both theories: Shannon entropy versus Kolmogorov complexity, the relation of both to universal coding, Shannon mutual information versus Kolmogorov (`algorithmic') mutual information, probabilistic sufficient statistic versus algorithmic sufficient statistic (related to lossy compression in the Shannon theory versus meaningful information in the Kolmogorov theory), and rate distortion theory versus Kolmogorov's structure function. Part of the material has appeared in print before, scattered through various publications, but this is the first comprehensive systematic comparison. The last mentioned relations are new.

Shannon Information and Kolmogorov Complexity

TL;DR

The paper juxtaposes Shannon entropy with Kolmogorov complexity to clarify their distinct aims: ensemble-based coding versus individual-object description. It shows that the expected Kolmogorov complexity under a computable distribution approximates Shannon entropy, and that probabilistic mutual information corresponds closely to its algorithmic analogue on average. By introducing the algorithmic structure function and algorithmic sufficiency, the authors connect MDL and rate-distortion theory to a purely descriptional framework, enabling model selection that balances meaningful structure with parsimony. The work also establishes that probabilistic and algorithmic notions of information obey analogous inequalities, with algorithmic quantities matching probabilistic ones up to additive (often logarithmic) terms, and it highlights practical pathways through universal coding and MDL as bridges between theory and data analysis.

Abstract

We compare the elementary theories of Shannon information and Kolmogorov complexity, the extent to which they have a common purpose, and where they are fundamentally different. We discuss and relate the basic notions of both theories: Shannon entropy versus Kolmogorov complexity, the relation of both to universal coding, Shannon mutual information versus Kolmogorov (`algorithmic') mutual information, probabilistic sufficient statistic versus algorithmic sufficient statistic (related to lossy compression in the Shannon theory versus meaningful information in the Kolmogorov theory), and rate distortion theory versus Kolmogorov's structure function. Part of the material has appeared in print before, scattered through various publications, but this is the first comprehensive systematic comparison. The last mentioned relations are new.

Paper Structure

This paper contains 76 sections, 24 theorems, 139 equations, 3 figures.

Table of Contents

  1. Introduction
  2. How to read this paper:
  3. Overview and Summary
  4. Preliminaries
  5. Strings:
  6. Precision and $\stackrel{{}_+}{<}, \stackrel{{}_+}{=}$ notation:
  7. Probabilistic Notions:
  8. Computable Functions:
  9. Codes
  10. Prefix code:
  11. Prefix codes and the Kraft inequality:
  12. Uniquely Decodable Codes:
  13. Probability distributions and complete prefix codes:
  14. Shannon Entropy versus Kolmogorov Complexity
  15. Shannon Entropy
  16. ...and 61 more sections

Key Result

Theorem 1.2

Let $l_1 , l_2 , \ldots$ be a finite or infinite sequence of natural numbers. There is a prefix-code with this sequence as lengths of its binary code words iff

Figures (3)

  • Figure 1: Structure functions $h_x(i), \beta_x(\alpha), \lambda_x(\alpha)$, and minimal sufficient statistic.
  • Figure 2: Data string $x$ is "positive random" or "stochastic" and data string $y$ is just "negative random" or "non-stochastic".
  • Figure 3: Structure function $h_x(i)= \min_f \{ \log 1/ f(x): f(x)>0, \; K(f) \leq i\}$ with $f$ a computable probability mass function, with values according to the left vertical coordinate, and the maximum likelihood estimator $2^{-h_x(i)}= \max \{f(x): p(x)>0 , \; K(f) \leq i\}$, with values according to the right-hand side vertical coordinate.

Theorems & Definitions (71)

  • Definition 1.1
  • Theorem 1.2
  • Definition 2.1
  • Theorem 2.2
  • Example 2.3
  • Definition 2.4
  • Theorem 2.5
  • Definition 2.6
  • Remark 2.7
  • Theorem 2.8
  • ...and 61 more