Generalized Maximum Entropy: When and Why you need it

TL;DR

The paper addresses when Shannon-based Maximum Entropy fails due to violations of strong system independence (SSI) and advocates using generalized entropies from the UJK family. It reviews the Shore–Johnson axioms from a practical viewpoint and situates common entropies (Shannon, Rényi, Tsallis) within the generalized framework, illustrated by economics and ecology case studies and a simple model that exhibits non-exponential phase-space growth. The authors provide a practical workflow for selecting entropy measures, estimating the governing parameter $q$, and reporting results to ensure transparency and reproducibility. By tying phase-space scaling to the appropriateness of SSI, the work offers a principled approach to applying ME frameworks to complex, correlated systems in science and economics.

Abstract

The classical Maximum-Entropy Principle (MEP) based on Shannon entropy is widely used to construct least-biased probability distributions from partial information. However, the Shore-Johnson axioms that single out the Shannon functional hinge on strong system independence, an assumption often violated in real-world, strongly correlated systems. We provide a self-contained guide to when and why practitioners should abandon the Shannon form in favour of the one-parameter Uffink-Jizba-Korbel (UJK) family of generalized entropies. After reviewing the Shore and Johnson axioms from an applied perspective, we recall the most commonly used entropy functionals and locate them within the UJK family. The need for generalized entropies is made clear with two applications, one rooted in economics and the other in ecology. A simple mathematical model worked out in detail shows the power of generalized maximum entropy approaches in dealing with cases where strong system independence does not hold. We conclude with practical guidelines for choosing an entropy measure and reporting results so that analyses remain transparent and reproducible.

Figures (1)

• Figure 1: (a) Histogram of observed $\Vec{M}^*$, together with inferred probability distribution $p(M|q^*,\psi^*)$. (b) contour plot of the log-likelihood surface, expressed as the difference $\ell(\hat{q},\hat{\psi})-\ell(q,\psi)$ between the maximum value (at the MLE $(\hat{q},\hat{\psi})$) and the value at each $(q,\psi)$.