Markov Categories and Entropy
Paolo Perrone
TL;DR
The work develops a unified framework that embeds quantitative information measures into Markov categories by enriching hom-sets with divergences and metrics. It shows how classical quantities such as $D_{KL}$, Rényi $D_\alpha$, and total-variation $d_T$ yield enrichments, enabling categorical definitions of mutual information $I_D$ and entropy $H_D$ that recover Shannon, Rényi, and Gini-Simpson (linear) forms, including extensions to nondiscrete and standard Borel settings. The paper provides equivalent characterizations via joints and marginals, derives data-processing inequalities in the enriched setting, and presents explicit formulas for mutual information and entropies in both discrete and continuous contexts, along with conditional variants. It also discusses limitations in continuous spaces and outlines future directions toward geometry-aware categorical information theory and entropy notions beyond standard measurable spaces.
Abstract
Markov categories are a novel framework to describe and treat problems in probability and information theory. In this work we combine the categorical formalism with the traditional quantitative notions of entropy, mutual information, and data processing inequalities. We show that several quantitative aspects of information theory can be captured by an enriched version of Markov categories, where the spaces of morphisms are equipped with a divergence or even a metric. As it is customary in information theory, mutual information can be defined as a measure of how far a joint source is from displaying independence of its components. More strikingly, Markov categories give a notion of determinism for sources and channels, and we can define entropy exactly by measuring how far a source or channel is from being deterministic. This recovers Shannon and Rényi entropies, as well as the Gini-Simpson index used in ecology to quantify diversity, and it can be used to give a conceptual definition of generalized entropy.