PAC-Bayesian Bounds on Constrained f-Entropic Risk Measures
Hind Atbir, Farah Cherfaoui, Guillaume Metzler, Emilie Morvant, Paul Viallard
TL;DR
The paper tackles generalization under subgroup imbalances by introducing constrained $f$-entropic risk measures, which extend CVaR via $f$-divergences to control distributional shifts. It develops both classical and disintegrated PAC-Bayesian bounds for these risks in two regimes, and shows they can be minimized with a self-bounding algorithm that provides subgroup-level guarantees. The theoretical bounds are instantiated with concrete deviation functions and accelerated by a practical optimization setup, while experiments on imbalanced OpenML datasets illustrate improved subgroup balance and competitive performance. This work advances reliable, subgroup-aware generalization guarantees and offers a practical route to bound-aware model learning.
Abstract
PAC generalization bounds on the risk, when expressed in terms of the expected loss, are often insufficient to capture imbalances between subgroups in the data. To overcome this limitation, we introduce a new family of risk measures, called constrained f-entropic risk measures, which enable finer control over distributional shifts and subgroup imbalances via f-divergences, and include the Conditional Value at Risk (CVaR), a well-known risk measure. We derive both classical and disintegrated PAC-Bayesian generalization bounds for this family of risks, providing the first disintegratedPAC-Bayesian guarantees beyond standard risks. Building on this theory, we design a self-bounding algorithm that minimizes our bounds directly, yielding models with guarantees at the subgroup level. Finally, we empirically demonstrate the usefulness of our approach.