Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Researcher's Guide to Empirical Risk Minimization

Lars van der Laan

Abstract

This guide develops high-probability regret bounds for empirical risk minimization (ERM). The presentation is modular: we state broadly applicable guarantees under high-level conditions and give tools for verifying them for specific losses and function classes. We emphasize that many ERM rate derivations can be organized around a three-step recipe -- a basic inequality, a uniform local concentration bound, and a fixed-point argument -- which yields regret bounds in terms of a critical radius, defined via localized Rademacher complexity, under a mild Bernstein-type variance--risk condition. To make these bounds concrete, we upper bound the critical radius using local maximal inequalities and metric-entropy integrals, recovering familiar rates for VC-subgraph, Sobolev/Hölder, and bounded-variation classes. We also review ERM with nuisance components -- including weighted ERM and Neyman-orthogonal losses -- as they arise in causal inference, missing data, and domain adaptation. Following the orthogonal learning framework, we highlight that these problems often admit regret-transfer bounds linking regret under an estimated loss to population regret under the target loss. These bounds typically decompose regret into (i) statistical error under the estimated (optimized) loss and (ii) approximation error due to nuisance estimation. Under sample splitting or cross-fitting, the first term can be controlled using standard fixed-loss ERM regret bounds, while the second term depends only on nuisance-estimation accuracy. We also treat the in-sample regime, where nuisances and the ERM are fit on the same data, deriving regret bounds and giving sufficient conditions for fast rates.

A Researcher's Guide to Empirical Risk Minimization

Abstract

This guide develops high-probability regret bounds for empirical risk minimization (ERM). The presentation is modular: we state broadly applicable guarantees under high-level conditions and give tools for verifying them for specific losses and function classes. We emphasize that many ERM rate derivations can be organized around a three-step recipe -- a basic inequality, a uniform local concentration bound, and a fixed-point argument -- which yields regret bounds in terms of a critical radius, defined via localized Rademacher complexity, under a mild Bernstein-type variance--risk condition. To make these bounds concrete, we upper bound the critical radius using local maximal inequalities and metric-entropy integrals, recovering familiar rates for VC-subgraph, Sobolev/Hölder, and bounded-variation classes. We also review ERM with nuisance components -- including weighted ERM and Neyman-orthogonal losses -- as they arise in causal inference, missing data, and domain adaptation. Following the orthogonal learning framework, we highlight that these problems often admit regret-transfer bounds linking regret under an estimated loss to population regret under the target loss. These bounds typically decompose regret into (i) statistical error under the estimated (optimized) loss and (ii) approximation error due to nuisance estimation. Under sample splitting or cross-fitting, the first term can be controlled using standard fixed-loss ERM regret bounds, while the second term depends only on nuisance-estimation accuracy. We also treat the in-sample regime, where nuisances and the ERM are fit on the same data, deriving regret bounds and giving sufficient conditions for fast rates.
Paper Structure (52 sections, 36 theorems, 335 equations, 2 tables)

This paper contains 52 sections, 36 theorems, 335 equations, 2 tables.

Table of Contents

  1. Introduction
  2. Scope and intent.
  3. Organization.
  4. Preliminaries
  5. Problem setup and notation
  6. High-probability guarantees (PAC bounds)
  7. From regret to $L^2(P)$ error: curvature and strong convexity
  8. Proof blueprint for regret bound
  9. Step 1: the basic inequality (deterministic regret bound)
  10. Warm-up: turning the basic inequality into rates
  11. From deterministic to random functions via uniform local concentration inequalities.
  12. A three-step template for ERM rates
  13. Step 2. Control randomness through a high-probability bound.
  14. Step 3. Bound regret and extract a rate.
  15. Remark.
  16. ...and 37 more sections

Key Result

Lemma 1

Suppose $\mathcal{F}$ is convex and $R:\mathcal{F}\to\mathbb R$ is strongly convex around $f_0$ with curvature constant $\kappa\in(0,\infty)$. Then, for all $f\in\mathcal{F}$,

Theorems & Definitions (59)

  • Lemma 1
  • proof
  • Example 1: Strong convexity for least squares via projection
  • Theorem 1: Basic inequality for constrained ERM
  • proof
  • Lemma 2: Bernstein bound for a fixed loss difference
  • Lemma 3: Sufficient conditions for the Bernstein condition
  • Lemma 4: Young's inequality
  • Theorem 2: Uniform local concentration inequality
  • Theorem 3: Regret bound for ERM
  • ...and 49 more