The Entropy and Crossentropy of Generalized Mallows Models
Marina Meilă
TL;DR
The paper addresses the challenge of information-theoretic analysis for the Generalized Mallows Model (GMM) over permutations by deriving tractable, closed-form-like expressions for entropy $H$, cross-entropy $\mathrm{XE}$, and KL divergence $D$ through the inversion matrix $Q$ and stagewise statistics $s_r$. It shows that the GMM forms an exponential family in the dispersion parameters $\vec{\theta}$ and that the discrete central permutation $\sigma$ can be handled via sufficient statistics, yielding a practical, polynomial-time recursion to compute necessary expectations $\bar{\mathbf Q}^{\rm up}_k(\sigma,\alpha_{1:r-1})$. The key contributions include a decomposition of $\mathrm{XE}$ into a sum over stages, an explicit expression for $H(\vec{\theta})$ independent of $\sigma$, and a recursive method to compute the averaged submatrices of $Q$, enabling efficient evaluation of divergence quantities. This work enables principled information-theoretic analysis and model comparison for ranking data within GMMs, with potential applications to ML and ranking-system design.
Abstract
The Generalized Mallows Model (GMM) is a well known family of models for ranking data. A GMM is a distribution over $\mathbb{S}_n$, the set of permutations of n objects, characterized by a location parameter $σ\in \mathbb{S}_n$, known as central permutation and a set of dispersion parameters $θ_{1:n-1}\in(0,1]$. The GMM shares many properties, such as having sufficient statistics, with exponential models, thus it can be seen as an exponential family with a discrete parameter $σ$. This paper shows that computing entropy, crossentropy and Kullback-Leibler divergence in the the class of GMM is tractable, paving the way for a better understanding of this exponential family.