Small Loss Bounds for Online Learning Separated Function Classes: A Gaussian Process Perspective
Adam Block, Abhishek Shetty
TL;DR
The paper addresses efficient learning in adversarial online and private-data settings by introducing ρ-separation, a condition that ensures sufficient distinguishability of functions under a separating measure. It unifies prior ideas such as small separator sets and γ-approximability, enabling oracle-efficient algorithms that achieve small-loss bounds and optimal private rates under separation, supported by a strengthened Gaussian-process stability result. The authors develop an online algorithm based on Follow the Perturbed Leader with Gaussian perturbations and provide a DP variant via Perturbed ERM, showing that stability under ρ-separation yields near-optimal performance in both settings. This framework offers practical, computationally tractable guarantees for adapting to favorable problem instances while preserving privacy and computational efficiency.
Abstract
In order to develop practical and efficient algorithms while circumventing overly pessimistic computational lower bounds, recent work has been interested in developing oracle-efficient algorithms in a variety of learning settings. Two such settings of particular interest are online and differentially private learning. While seemingly different, these two fields are fundamentally connected by the requirement that successful algorithms in each case satisfy stability guarantees; in particular, recent work has demonstrated that algorithms for online learning whose performance adapts to beneficial problem instances, attaining the so-called small-loss bounds, require a form of stability similar to that of differential privacy. In this work, we identify the crucial role that separation plays in allowing oracle-efficient algorithms to achieve this strong stability. Our notion, which we term $ρ$-separation, generalizes and unifies several previous approaches to enforcing this strong stability, including the existence of small-separator sets and the recent notion of $γ$-approximability. We present an oracle-efficient algorithm that is capable of achieving small-loss bounds with improved rates in greater generality than previous work, as well as a variant for differentially private learning that attains optimal rates, again under our separation condition. In so doing, we prove a new stability result for minimizers of a Gaussian process that strengthens and generalizes previous work.