Sharp Rates in Dependent Learning Theory: Avoiding Sample Size Deflation for the Square Loss

TL;DR

This work addresses statistical learning with dependent ($β$-mixing) data under the square loss, aiming for sharp interaction terms that capture noise without deflating the effective sample size by mixing. It develops a framework that combines weakly sub-Gaussian $Ψ_p$-norm control with mixed-tail generic chaining to bound two key empirical processes: the multiplier process and the quadratic process, before extending to $β$-mixing data via blocking. The main result shows an instance-optimal rate characterized by a fixed-point radius $r_\star$, depending on the local $L^2$-complexity via the $\gamma_2$-functional and a second-order variance proxy $\mathbf{V}_{2q}$, with mixing affecting only higher-order burn-in terms. The framework applies to sub-Gaussian linear regression, smoothly parameterized functions, finite hypothesis classes, and other weakly smooth classes, offering sharp, interpretable rates in dependent settings and improving upon prior deflation-based analyses.

Abstract

In this work, we study statistical learning with dependent ($β$-mixing) data and square loss in a hypothesis class $\mathscr{F}\subset L_{Ψ_p}$ where $Ψ_p$ is the norm $\|f\|_{Ψ_p} \triangleq \sup_{m\geq 1} m^{-1/p} \|f\|_{L^m} $ for some $p\in [2,\infty]$. Our inquiry is motivated by the search for a sharp noise interaction term, or variance proxy, in learning with dependent data. Absent any realizability assumption, typical non-asymptotic results exhibit variance proxies that are deflated multiplicatively by the mixing time of the underlying covariates process. We show that whenever the topologies of $L^2$ and $Ψ_p$ are comparable on our hypothesis class $\mathscr{F}$ -- that is, $\mathscr{F}$ is a weakly sub-Gaussian class: $\|f\|_{Ψ_p} \lesssim \|f\|_{L^2}^η$ for some $η\in (0,1]$ -- the empirical risk minimizer achieves a rate that only depends on the complexity of the class and second order statistics in its leading term. Our result holds whether the problem is realizable or not and we refer to this as a \emph{near mixing-free rate}, since direct dependence on mixing is relegated to an additive higher order term. We arrive at our result by combining the above notion of a weakly sub-Gaussian class with mixed tail generic chaining. This combination allows us to compute sharp, instance-optimal rates for a wide range of problems. Examples that satisfy our framework include sub-Gaussian linear regression, more general smoothly parameterized function classes, finite hypothesis classes, and bounded smoothness classes.

Theorems & Definitions (40)

• Lemma 2.1: $\Psi_p$-Bernstein MGF Bound
• Definition 2.1: Weakly sub-Gaussian Class
• Definition 2.2: Noise Level
• Lemma 2.2: Pointwise Control
• Theorem 2.1
• Theorem 2.2: Lower Uniform Law
• Definition 2.3
• Theorem 3.1
• Corollary 3.1: Parametric Classes
• Corollary 3.2: Realizable Linear Regression