Universal Batch Learning Under The Misspecification Setting

TL;DR

This work tackles universal batch learning under misspecification with log-loss, where data are generated from a distribution set $\Phi$ larger than the hypothesis class $\Theta$. It derives a closed-form min-max regret $R^*_N(\Theta,\Phi)$ as $\max_{\pi(\phi)} [ I(Y_N;\Phi|Y^{N-1}) - E_{\pi} { D_{c,N}(P_\phi\|\Theta) } ]$ and introduces a mixture prior $\pi(\phi)$ that induces a capacity-like universal predictor $Q_{\pi}$. The authors develop an Arimoto-Blahut extension to numerically evaluate the regret and demonstrate the theory on Bernoulli/multinomial settings, including bounds and extensions to combined batch-online and supervised batch learning. A key finding is that the regret is governed by the richness of the hypothesis set $\Theta$ rather than the full generating set $\Phi$, and that the mixture concentrates mass near $\Theta$, providing a principled approach for robust universal learning under misspecification. The framework lays groundwork for practical computation of capacity-like priors and worst-case regrets in agnostic data-generation scenarios, with potential impact on robust universal predictors.

Abstract

In this paper we consider the problem of universal {\em batch} learning in a misspecification setting with log-loss. In this setting the hypothesis class is a set of models $Θ$. However, the data is generated by an unknown distribution that may not belong to this set but comes from a larger set of models $Φ\supset Θ$. Given a training sample, a universal learner is requested to predict a probability distribution for the next outcome and a log-loss is incurred. The universal learner performance is measured by the regret relative to the best hypothesis matching the data, chosen from $Θ$. Utilizing the minimax theorem and information theoretical tools, we derive the optimal universal learner, a mixture over the set of the data generating distributions, and get a closed form expression for the min-max regret. We show that this regret can be considered as a constrained version of the conditional capacity between the data and its generating distributions set. We present tight bounds for this min-max regret, implying that the complexity of the problem is dominated by the richness of the hypotheses models $Θ$ and not by the data generating distributions set $Φ$. We develop an extension to the Arimoto-Blahut algorithm for numerical evaluation of the regret and its capacity achieving prior distribution. We demonstrate our results for the case where the observations come from a $K$-parameters multinomial distributions while the hypothesis class $Θ$ is only a subset of this family of distributions.

Figures (3)

• Figure 1: Theorem \ref{['Thm2']}
• Figure 2: Stochastic vs. Misspecified Capacity Achieving Prior distribution for settings: (a), (b) and (c), where $N=10^3.$
• Figure 3: Stochastic vs. Misspecified $\beta$ bias factor for settings: (a), (b) and (c), where $N=10^2$.

Theorems & Definitions (15)

• Theorem 1
• proof
• Lemma 1
• proof
• Theorem 2
• proof
• Theorem 3
• proof
• Theorem 4
• Theorem 5