Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Typicality, entropy and the generalization of statistical mechanics

Bernat Corominas-Murtra, Rudolf Hanel, Petr Jizba

TL;DR

The paper investigates typicality as a unifying principle behind thermodynamics and information theory, using a toy model of a thermalized coin to illustrate how the typical set captures almost all probability with a small state-space footprint. It shows that Shannon entropy naturally characterizes typical-set occupancy, while Rényi and Tsallis entropies lead to thermodynamic-like constructs such as free energy and the partition function, respectively. The authors then extend the framework beyond i.i.d. processes to Compact Stochastic Processes (CSPs), introducing a generalized entropy S_\Lambda that governs the typical-set size even when the phase space grows or contracts. These results point toward a generalized statistical mechanics for complex systems and raise open questions about connections to coarse-graining, large deviations, and non-i.i.d. dynamics.

Abstract

When at equilibrium, large-scale systems obey conventional thermodynamics because they belong to microscopic configurations (or states) that are typical. Crucially, the typical states usually represent only a small fraction of the total number of possible states, and yet the characterization of the set of typical states -- the typical set -- alone is sufficient to describe the macroscopic behavior of a given system. Consequently, the concept of typicality, and the associated Asymptotic Equipartition Property allow for a drastic reduction of the degrees of freedom needed for system's statistical description. The mathematical rationale for such a simplification in the description is due to the phenomenon of concentration of measure. The later emerges for equilibrium configurations thanks to very strict constraints on the underlying dynamics, such as weekly interacting and (almost) independent system constituents. The question naturally arises as to whether the concentration of measure and related typicality considerations can be extended and applied to more general complex systems, and if so, what mathematical structure can be expected in the ensuing generalized thermodynamics. In this paper we illustrate the relevance of the concept of typicality in the toy model context of the "thermalized" coin and show how this leads naturally to Shannon entropy. We also show an intriguing connection: The characterization of typical sets in terms of Renyi and Tsallis entropies naturally leads to the free energy and partition function, respectively, and makes their relationship explicit. Finally, we propose potential ways to generalize the concept of typicality to systems where the standard microscopic assumptions do not hold.

Typicality, entropy and the generalization of statistical mechanics

TL;DR

The paper investigates typicality as a unifying principle behind thermodynamics and information theory, using a toy model of a thermalized coin to illustrate how the typical set captures almost all probability with a small state-space footprint. It shows that Shannon entropy naturally characterizes typical-set occupancy, while Rényi and Tsallis entropies lead to thermodynamic-like constructs such as free energy and the partition function, respectively. The authors then extend the framework beyond i.i.d. processes to Compact Stochastic Processes (CSPs), introducing a generalized entropy S_\Lambda that governs the typical-set size even when the phase space grows or contracts. These results point toward a generalized statistical mechanics for complex systems and raise open questions about connections to coarse-graining, large deviations, and non-i.i.d. dynamics.

Abstract

When at equilibrium, large-scale systems obey conventional thermodynamics because they belong to microscopic configurations (or states) that are typical. Crucially, the typical states usually represent only a small fraction of the total number of possible states, and yet the characterization of the set of typical states -- the typical set -- alone is sufficient to describe the macroscopic behavior of a given system. Consequently, the concept of typicality, and the associated Asymptotic Equipartition Property allow for a drastic reduction of the degrees of freedom needed for system's statistical description. The mathematical rationale for such a simplification in the description is due to the phenomenon of concentration of measure. The later emerges for equilibrium configurations thanks to very strict constraints on the underlying dynamics, such as weekly interacting and (almost) independent system constituents. The question naturally arises as to whether the concentration of measure and related typicality considerations can be extended and applied to more general complex systems, and if so, what mathematical structure can be expected in the ensuing generalized thermodynamics. In this paper we illustrate the relevance of the concept of typicality in the toy model context of the "thermalized" coin and show how this leads naturally to Shannon entropy. We also show an intriguing connection: The characterization of typical sets in terms of Renyi and Tsallis entropies naturally leads to the free energy and partition function, respectively, and makes their relationship explicit. Finally, we propose potential ways to generalize the concept of typicality to systems where the standard microscopic assumptions do not hold.
Paper Structure (14 sections, 99 equations, 1 figure)

This paper contains 14 sections, 99 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Concentration of measure: tossing a coin
  3. A coin's "microcanonical" ensemble
  4. A coin's "canonical" ensemble
  5. The typical set
  6. "Temperature" and occupation of the state space
  7. Typicality in coin tossing systems --- going beyond Shannon's paradigm
  8. Typical set from Rényi entropy
  9. Typical sets from Tsallis entropy
  10. Entropy and typicality: going beyond i.i.d.
  11. Basics: CSP's
  12. The typical set and entropy in CSP's
  13. Discussion and Conclusions
  14. Kolmogorov--Nagumo mean and Rényi-type typical set

Figures (1)

  • Figure 1: A/ Different underlying microscopic hypothesis: (Top), the standard i.i.d. assumption, or its physical analogue, the Stosszahlansatz or chaos molecular hypothesis. (Bottom) a stochastic process whose sampling space grows in time, thereby violating the assumptions over which standard statistical mechanics is built. B/ (Top) A physical system in equilibrium whose microscopic dynamics obeys the molecular-chaos hypothesis and (Bottom) a toy representation of the collective behaviour of a physical system with increasing sampling space. C/ (Top) The standard assumptions of equilibrium statistical mechanics lead naturally to the concept of typical set. (Bottom) more complex dynamics may also have typical behaviours, albeit more complex to identify or characterize. D/ (Top) The existence of the typical set in equilibrium configurations gives rise to the Shannon entropy, as the natural functional accounting for the typical occupation of the sampling space. (Bottom) more complex dynamics still leading to typical behaviours may give rise to generalized forms of entropy and other functionals.