Confidence Sequences for Generalized Linear Models via Regret Analysis
Eugenio Clerico, Hamish Flynn, Wojciech Kotłowski, Gergely Neu
TL;DR
This work develops a unifying framework that translates confidence-set construction for generalized linear models into regret guarantees for sequential prediction. By two key routes—analytic conversions that bound confidence-set widths via regret and algorithmic conversions that exploit online predictions—the authors derive tight, data-dependent confidence sequences and new types of sets, including ones invariant to covariate scaling and independent of sample size in certain regimes. The main technical tools are sequential likelihood ratios, Ville’s inequality, exponentially weighted average (EWA) forecasters, and normalized maximum likelihood (NML), connected through a notion of Bregman information gain. The results recover or improve state-of-the-art bounds for GLMs under various covariate regimes (adaptive, oblivious) and extend to sparse settings, while offering a principled pathway to extending uncertainty quantification beyond GLMs. The framework thus links statistical inference with online learning, enabling systematic derivation of concentration bounds via regret analysis and suggesting avenues for future work in non-convex models and broader distribution families.
Abstract
We develop a methodology for constructing confidence sets for parameters of statistical models via a reduction to sequential prediction. Our key observation is that for any generalized linear model (GLM), one can construct an associated game of sequential probability assignment such that achieving low regret in the game implies a high-probability upper bound on the excess likelihood of the true parameter of the GLM. This allows us to develop a scheme that we call online-to-confidence-set conversions, which effectively reduces the problem of proving the desired statistical claim to an algorithmic question. We study two varieties of this conversion scheme: 1) analytical conversions that only require proving the existence of algorithms with low regret and provide confidence sets centered at the maximum-likelihood estimator 2) algorithmic conversions that actively leverage the output of the online algorithm to construct confidence sets (and may be centered at other, adaptively constructed point estimators). The resulting methodology recovers all state-of-the-art confidence set constructions within a single framework, and also provides several new types of confidence sets that were previously unknown in the literature.