Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Alpha-NML Universal Predictors

Marco Bondaschi, Michael Gastpar

TL;DR

The paper introduces alpha-NML predictors, a parametric universal prediction family indexed by alpha >= 1 that interpolate between mixture estimators and the Normalized Maximum Likelihood, grounded in regret measures linked to Rényi divergence. It proves optimality of alpha-NML under alpha-regret with an appropriately chosen prior that maximizes Sibson's alpha mutual information, and discusses existence conditions and connections to Luckiness NML, including cases where NML does not exist. The authors specialize to discrete memoryless sources, deriving tractable closed-form expressions and asymptotic worst-case regret that show alpha-NML interpolates between the KT estimator and NML, while preserving computational efficiency for integer alpha. The work provides a flexible, robust prediction framework with theoretical guarantees and practical formulas, expanding the universal prediction toolbox to settings where NML is problematic or inapplicable. Overall, alpha-NML offers a tunable balance between average and worst-case performance with concrete benefits for DMS and potential broader applicability in universal compression and learning.

Abstract

Inspired by the connection between classical regret measures employed in universal prediction and Rényi divergence, we introduce a new class of universal predictors that depend on a real parameter $α\geq 1$. This class interpolates two well-known predictors, the mixture estimators, that include the Laplace and the Krichevsky-Trofimov predictors, and the Normalized Maximum Likelihood (NML) estimator. We point out some advantages of this new class of predictors and study its benefits from two complementary viewpoints: (1) we prove its optimality when the maximal Rényi divergence is considered as a regret measure, which can be interpreted operationally as a middle ground between the standard average and worst-case regret measures; (2) we discuss how it can be employed when NML is not a viable option, as an alternative to other predictors such as Luckiness NML. Finally, we apply the $α$-NML predictor to the class of discrete memoryless sources (DMS), where we derive simple formulas to compute the predictor and analyze its asymptotic performance in terms of worst-case regret.

Alpha-NML Universal Predictors

TL;DR

The paper introduces alpha-NML predictors, a parametric universal prediction family indexed by alpha >= 1 that interpolate between mixture estimators and the Normalized Maximum Likelihood, grounded in regret measures linked to Rényi divergence. It proves optimality of alpha-NML under alpha-regret with an appropriately chosen prior that maximizes Sibson's alpha mutual information, and discusses existence conditions and connections to Luckiness NML, including cases where NML does not exist. The authors specialize to discrete memoryless sources, deriving tractable closed-form expressions and asymptotic worst-case regret that show alpha-NML interpolates between the KT estimator and NML, while preserving computational efficiency for integer alpha. The work provides a flexible, robust prediction framework with theoretical guarantees and practical formulas, expanding the universal prediction toolbox to settings where NML is problematic or inapplicable. Overall, alpha-NML offers a tunable balance between average and worst-case performance with concrete benefits for DMS and potential broader applicability in universal compression and learning.

Abstract

Inspired by the connection between classical regret measures employed in universal prediction and Rényi divergence, we introduce a new class of universal predictors that depend on a real parameter . This class interpolates two well-known predictors, the mixture estimators, that include the Laplace and the Krichevsky-Trofimov predictors, and the Normalized Maximum Likelihood (NML) estimator. We point out some advantages of this new class of predictors and study its benefits from two complementary viewpoints: (1) we prove its optimality when the maximal Rényi divergence is considered as a regret measure, which can be interpreted operationally as a middle ground between the standard average and worst-case regret measures; (2) we discuss how it can be employed when NML is not a viable option, as an alternative to other predictors such as Luckiness NML. Finally, we apply the -NML predictor to the class of discrete memoryless sources (DMS), where we derive simple formulas to compute the predictor and analyze its asymptotic performance in terms of worst-case regret.
Paper Structure (21 sections, 6 theorems, 93 equations, 1 figure)

This paper contains 21 sections, 6 theorems, 93 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related work
  4. Overview
  5. Notation
  6. $\alpha$-regret
  7. $\alpha$-NML predictors
  8. Definition
  9. Optimality with respect to $\alpha$-regret
  10. $\alpha$-NML as an alternative to NML
  11. Existence of $\alpha$-NML and choice of $w$ and $\alpha$
  12. Worst-case regret of $\alpha$-NML
  13. Connection between $\alpha$-NML and Luckiness NML
  14. $\alpha$-NML for DMS
  15. The Krichevsky-Trofimov estimator
  16. ...and 6 more sections

Key Result

Theorem 1

Let $\mathcal{X}$ be a (possibly infinite) alphabet set, and let $\mathcal{P} = \{p_{\theta} : \theta\in\Theta\}$ be a parametric class of distributions on $\mathcal{X}^n$. Let $(\phi, X^n) \sim w \times p_{\phi}(X^n)$ for some prior distribution $w$ on $\Theta$, and suppose that there exists a prob Then, the $\alpha$-NML defined in AlphaNML with prior $w^*$, i.e., minimizes $R_{\alpha}(\hat{p})$

Figures (1)

  • Figure 1: Percentage of increase of $R_{\max}(\hat{p}_{\alpha})$ with respect to the optimal value $R_{\max}(\hat{p}_{\rm NML})$, as a function of $\alpha$, for binary sequences of length $n=10,50,100$ and integer values of $\alpha$. The value at $\alpha = 1$ corresponds to the regret of the Krichevsky-Trofimov estimator.

Theorems & Definitions (11)

  • Definition 1
  • Definition 2
  • Theorem 1
  • Theorem 2
  • Example 1
  • Example 2
  • Example 3
  • Lemma 1
  • Theorem 3
  • Theorem 4
  • ...and 1 more