Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

High-Probability Minimax Adaptive Estimation in Besov Spaces via Online-to-Batch

Paul Liautaud, Pierre Gaillard, Olivier Wintenberger

TL;DR

This work tackles nonparametric regression in Besov spaces under sub-exponential noise, aiming for high-probability minimax-adaptive risk bounds that are unknown-noise–robust. It introduces a wavelet-based online learning framework with adaptive gradient clipping, enabling comparator-adaptive regret bounds and a refined online-to-batch conversion to achieve minimax rates without prior knowledge of the noise level. The main contributions are high-probability regret results with both known and adaptive clipping, an adaptive clipping strategy via expert aggregation, and a multiscale online wavelet regression procedure that attains noise-level–aware minimax risk bounds in high probability for Besov-function regression. The approach yields minimax-optimal rates, matching recent noise-aware batch results, while providing practical adaptivity and computational efficiency, with potential impact on robust nonparametric estimation in practice.

Abstract

We study nonparametric regression over Besov spaces from noisy observations under sub-exponential noise, aiming to achieve minimax-optimal guarantees on the integrated squared error that hold with high probability and adapt to the unknown noise level. To this end, we propose a wavelet-based online learning algorithm that dynamically adjusts to the observed gradient noise by adaptively clipping it at an appropriate level, eliminating the need to tune parameters such as the noise variance or gradient bounds. As a by-product of our analysis, we derive high-probability adaptive regret bounds that scale with the $\ell_1$-norm of the competitor. Finally, in the batch statistical setting, we obtain adaptive and minimax-optimal estimation rates for Besov spaces via a refined online-to-batch conversion. This approach carefully exploits the structure of the squared loss in combination with self-normalized concentration inequalities.

High-Probability Minimax Adaptive Estimation in Besov Spaces via Online-to-Batch

TL;DR

This work tackles nonparametric regression in Besov spaces under sub-exponential noise, aiming for high-probability minimax-adaptive risk bounds that are unknown-noise–robust. It introduces a wavelet-based online learning framework with adaptive gradient clipping, enabling comparator-adaptive regret bounds and a refined online-to-batch conversion to achieve minimax rates without prior knowledge of the noise level. The main contributions are high-probability regret results with both known and adaptive clipping, an adaptive clipping strategy via expert aggregation, and a multiscale online wavelet regression procedure that attains noise-level–aware minimax risk bounds in high probability for Besov-function regression. The approach yields minimax-optimal rates, matching recent noise-aware batch results, while providing practical adaptivity and computational efficiency, with potential impact on robust nonparametric estimation in practice.

Abstract

We study nonparametric regression over Besov spaces from noisy observations under sub-exponential noise, aiming to achieve minimax-optimal guarantees on the integrated squared error that hold with high probability and adapt to the unknown noise level. To this end, we propose a wavelet-based online learning algorithm that dynamically adjusts to the observed gradient noise by adaptively clipping it at an appropriate level, eliminating the need to tune parameters such as the noise variance or gradient bounds. As a by-product of our analysis, we derive high-probability adaptive regret bounds that scale with the -norm of the competitor. Finally, in the batch statistical setting, we obtain adaptive and minimax-optimal estimation rates for Besov spaces via a refined online-to-batch conversion. This approach carefully exploits the structure of the squared loss in combination with self-normalized concentration inequalities.
Paper Structure (47 sections, 10 theorems, 98 equations, 2 algorithms)

This paper contains 47 sections, 10 theorems, 98 equations, 2 algorithms.

Table of Contents

  1. Introduction
  2. Notations.
  3. Contributions and outline of the paper.
  4. Related work
  5. Nonparametric regression over Besov spaces.
  6. Online-to-Batch.
  7. Online convex optimization with stochastic unbounded gradients.
  8. High probability comparator-adaptive regret with constrained iterates and noisy unbounded gradients
  9. Stochastic directional derivative condition.
  10. Algorithm: Adaptive Learning with Unbounded Noisy Gradients via Clipping
  11. Assumption on the coordinate subroutine.
  12. First result: high probability comparator-adaptive regret with known noise level.
  13. Adaptive clipping strategy
  14. Adaptive clipping via expert aggregation.
  15. Assumption on the expert aggregation algorithm.
  16. ...and 32 more sections

Key Result

Theorem 1

Let $G,C>0$, $\delta\in(0,1)$, and $N\in\mathbb N^*$. Assume $(\ell_t)_{t \geqslant 1}$ satisfy Assumption assump:exp-concave and at each time step $t\geqslant 1$ we are given a stochastic gradient $\nabla \ell_t(\mathbf{c}_t) \in \mathbb R^N$ satisfying Assumption assump:subexp with parameter $\nu, Then Algorithm alg:param_free_unbounded_gradients, run over $T$ steps with $C_n = C$, $G_{n,t} = G$

Theorems & Definitions (18)

  • Theorem 1: High-probability $\ell_1$-comparator-adaptive regret
  • Theorem 2
  • Theorem 3
  • Theorem 4: High-probability parameter-free regret
  • proof
  • Theorem 5
  • proof : of Theorem \ref{['theo:adapt_clipping_param_free_general']}
  • proof : of Theorem \ref{['theo:non_param_risk']}
  • Lemma 1: Exponential inequality
  • proof
  • ...and 8 more