Generalization bounds for mixing processes via delayed online-to-PAC conversions
Baptiste Abeles, Eugenio Clerico, Gergely Neu
TL;DR
This work addresses the challenge of bounding generalization error when training data come from a stationary mixing process, i.e., non-i.i.d. data with temporal dependence. It develops a general online-to-PAC conversion framework with delayed feedback, combining Yu’s blocking and the Lugosi online-to-PAC approach under a weak $\phi_d$-mixing assumption to yield high-probability generalization bounds. The authors derive explicit bounds for geometrically and algebraically mixing processes, and extend the framework to delayed variants of EWA and FTRL, as well as to dynamic hypotheses whose loss depends on memory. The results provide near-i.i.d.-like rates in several regimes, offer PAC-Bayesian-style guarantees for dependent data, and apply to time-series and dynamical-system predictors, broadening the scope of robust generalization analysis beyond i.i.d. settings.
Abstract
We study the generalization error of statistical learning algorithms in a non-i.i.d. setting, where the training data is sampled from a stationary mixing process. We develop an analytic framework for this scenario based on a reduction to online learning with delayed feedback. In particular, we show that the existence of an online learning algorithm with bounded regret (against a fixed statistical learning algorithm in a specially constructed game of online learning with delayed feedback) implies low generalization error of said statistical learning method even if the data sequence is sampled from a mixing time series. The rates demonstrate a trade-off between the amount of delay in the online learning game and the degree of dependence between consecutive data points, with near-optimal rates recovered in a number of well-studied settings when the delay is tuned appropriately as a function of the mixing time of the process.