Expressing entropy and cross-entropy in expansions of common meadows
Jan A Bergstra, John V Tucker
TL;DR
The paper tackles the problem of expressing entropy and cross-entropy as single equational terms over totalised real algebras, specifically common meadows augmented with a total logarithm. It demonstrates that the base algebra $\mathbb{R}_{\bot,\log_2}$ cannot define these measures for small sample spaces (e.g., $|V|=2$) and so investigates expansions with auxiliary operators such as conditional, left-sequential multiplication, and a sign function, achieving term-definability for all finite sizes when using $\mathsf{s}^2$. It further analyzes fracterm flattening, introduces alternative infinities, and connects the framework to KL/JS divergences and Bayes-Price reasoning, outlining implications for probabilistic reasoning in algebraic data types. The work provides a principled, algebraic route to complete information measures without case-splitting.
Abstract
A common meadow is an enrichment of a field with a partial division operation that is made total by assuming that division by zero takes the a default value, a special element $\bot$ adjoined to the field. To a common meadow of real numbers we add a binary logarithm $\log_2(-)$, which we also assume to be total with $\log_2(p) = \bot$ for $p \leq 0$. With these and other auxiliary operations, such as a sign function, we form algebras over which entropy and cross entropy can be defined for probability mass functions on a finite sample space by algebraic formulae that are simple terms built from the operations of the algebras and without case distinctions or conventions to avoid partiality. The discuss the advantages of algebras based on common meadows, whose theory is established, and alternate methods to define entropy and other information measures completely for all arguments using single terms.