Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayes
Charles Riou, Pierre Alquier, Badr-Eddine Chérief-Abdellatif
TL;DR
This paper studies meta-learning of priors through PAC-Bayes bounds in a two-level Gibbs-posterior framework. It establishes that Bernstein's condition automatically holds at the meta level, enabling fast rates in the number of tasks $T$ for meta-learning priors, with costs that scale as $O(1/T)$ and improvements in discrete, Gaussian, and mixtures-of-Gaussians priors. The authors derive a meta-learning PAC-Bayes bound, illustrate a toy concurrent-priors scenario, and provide explicit rate results across three prior families, including a favorable $O(\log T / T)$ regime under concentration. They discuss connections to existing theory, the potential for extending to variational approximations, and open questions for broader priors in meta-learning.
Abstract
Bernstein's condition is a key assumption that guarantees fast rates in machine learning. For example, the Gibbs algorithm with prior $π$ has an excess risk in $O(d_π/n)$, as opposed to the standard $O(\sqrt{d_π/n})$, where $n$ denotes the number of observations and $d_π$ is a complexity parameter which depends on the prior $π$. In this paper, we examine the Gibbs algorithm in the context of meta-learning, i.e., when learning the prior $π$ from $T$ tasks (with $n$ observations each) generated by a meta distribution. Our main result is that Bernstein's condition always holds at the meta level, regardless of its validity at the observation level. This implies that the additional cost to learn the Gibbs prior $π$, which will reduce the term $d_π$ across tasks, is in $O(1/T)$, instead of the expected $O(1/\sqrt{T})$. We further illustrate how this result improves on standard rates in three different settings: discrete priors, Gaussian priors and mixture of Gaussians priors.