A categorical characterization of relative entropy on standard Borel spaces
Nicolas Gagne, Prakash Panangaden
TL;DR
The paper extends a categorical treatment of relative entropy from finite sets to standard Borel spaces by leveraging the Giry monad and Rohlin disintegration. It defines a relative-entropy functor $RE:\mathbf{SbStat}\to[0,\infty]$ and proves its convex linearity, lower semicontinuity, and a uniqueness result up to a scalar, thereby generalizing the finite-case $RE_{fin}$ to the measure-theoretic setting. It further shows that $RE$ coincides with the Kullback–Leibler divergence on absolutely coherent morphisms and provides a principled foundation for entropy-based reasoning in learning and Bayesian inversion on standard Borel spaces. This framework clarifies how entropy can be incorporated into learning-theoretic constructions beyond finite domains and sets the stage for future point-free extensions.
Abstract
We give a categorical treatment, in the spirit of Baez and Fritz, of relative entropy for probability distributions defined on standard Borel spaces. We define a category suitable for reasoning about statistical inference on standard Borel spaces. We define relative entropy as a functor into Lawvere's category and we show convexity, lower semicontinuity and uniqueness.