Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Variance, Admissibility, and Stability of Empirical Risk Minimization

Gil Kur, Eli Putterman, Alexander Rakhlin

TL;DR

This work analyzes Empirical Risk Minimization (ERM) in convex function classes under both fixed-design and random-design regression. It shows that, under mild assumptions, the ERM variance is bounded by the minimax rate, implying that suboptimal ERM performance must stem from bias. The authors develop two complementary routes to bound the variance in the random-design setting: an empirical-process approach under uniform boundedness and an isoperimetric approach via Lipschitz concentration, with corresponding results on admissibility and stability that extend to non-Donsker classes. They also reveal irregular loss landscapes in the non-Donsker regime, demonstrating that near-ERM functions can be far from near-minimizers, highlighting challenges in optimization geometry. Collectively, the results connect variance control to minimax theory and provide principled insights into when ERM is (nearly) optimal, and when debiasing or robust choices are essential for large function classes.

Abstract

It is well known that Empirical Risk Minimization (ERM) may attain minimax suboptimal rates in terms of the mean squared error (Birgé and Massart, 1993). In this paper, we prove that, under relatively mild assumptions, the suboptimality of ERM must be due to its large bias. Namely, the variance error term of ERM is bounded by the minimax rate. In the fixed design setting, we provide an elementary proof of this result using the probabilistic method. Then, we extend our proof to the random design setting for various models. In addition, we provide a simple proof of Chatterjee's admissibility theorem (Chatterjee, 2014, Theorem 1.4), which states that in the fixed design setting, ERM cannot be ruled out as an optimal method, and then we extend this result to the random design setting. We also show that our estimates imply the stability of ERM, complementing the main result of Caponnetto and Rakhlin (2006) for non-Donsker classes. Finally, we highlight the somewhat irregular nature of the loss landscape of ERM in the non-Donsker regime, by showing that functions can be close to ERM, in terms of $L_2$ distance, while still being far from almost-minimizers of the empirical loss.

On the Variance, Admissibility, and Stability of Empirical Risk Minimization

TL;DR

This work analyzes Empirical Risk Minimization (ERM) in convex function classes under both fixed-design and random-design regression. It shows that, under mild assumptions, the ERM variance is bounded by the minimax rate, implying that suboptimal ERM performance must stem from bias. The authors develop two complementary routes to bound the variance in the random-design setting: an empirical-process approach under uniform boundedness and an isoperimetric approach via Lipschitz concentration, with corresponding results on admissibility and stability that extend to non-Donsker classes. They also reveal irregular loss landscapes in the non-Donsker regime, demonstrating that near-ERM functions can be far from near-minimizers, highlighting challenges in optimization geometry. Collectively, the results connect variance control to minimax theory and provide principled insights into when ERM is (nearly) optimal, and when debiasing or robust choices are essential for large function classes.

Abstract

It is well known that Empirical Risk Minimization (ERM) may attain minimax suboptimal rates in terms of the mean squared error (Birgé and Massart, 1993). In this paper, we prove that, under relatively mild assumptions, the suboptimality of ERM must be due to its large bias. Namely, the variance error term of ERM is bounded by the minimax rate. In the fixed design setting, we provide an elementary proof of this result using the probabilistic method. Then, we extend our proof to the random design setting for various models. In addition, we provide a simple proof of Chatterjee's admissibility theorem (Chatterjee, 2014, Theorem 1.4), which states that in the fixed design setting, ERM cannot be ruled out as an optimal method, and then we extend this result to the random design setting. We also show that our estimates imply the stability of ERM, complementing the main result of Caponnetto and Rakhlin (2006) for non-Donsker classes. Finally, we highlight the somewhat irregular nature of the loss landscape of ERM in the non-Donsker regime, by showing that functions can be close to ERM, in terms of distance, while still being far from almost-minimizers of the empirical loss.
Paper Structure (42 sections, 24 theorems, 212 equations)

This paper contains 42 sections, 24 theorems, 212 equations.

Table of Contents

  1. Introduction
  2. Prior Work
  3. Stability of ERM
  4. Shape-constrained regression
  5. High-dimensional statistics
  6. Main Results
  7. Preliminaries
  8. Notation
  9. Notation
  10. Fixed design setting
  11. Random design setting
  12. Bounding the variance via empirical processes approach
  13. Bounding the variance via isoperimetry approach
  14. Admissibility
  15. On the landscape of ERM in the non-Donsker regime
  16. ...and 27 more sections

Key Result

theorem 1

Under Assumptions A:Convex-A:normal, for any $f^{*} \in \mathcal{F}$, the following holds: and in particular $\mathcal{V}(\widehat{f}_n,\mathcal{F},\mathbb{P}^{(n)}) \lesssim \mathcal{M}(\mathcal{F},\mathbb{P}^{(n)})$. Furthermore, for $\delta:= \delta(f^{*},n) \lesssim \mathcal{M}(\mathcal{H}_{*},\mathbb{P}^{(n)})$, the event holds with probability at least $\max\{1 - 2\exp(-cn \cdot \mathcal{M

Theorems & Definitions (56)

  • definition 1
  • theorem 1
  • proposition 1
  • theorem 2: Chatterjee's Admissibility Theorem
  • remark 1
  • remark 2
  • remark 3
  • remark 4
  • definition 2
  • remark 5
  • ...and 46 more