Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Policy Learning via a Self-Normalized Maximal Inequality

Samuel Girard, Aurélien Bibaut, Houssam Zenati

TL;DR

The paper tackles policy learning with adaptively collected data, where dependence undermines classical i.i.d. guarantees. It develops a self-normalized maximal inequality for martingale empirical processes and builds Adaptive Sample Variance Penalization (ASVP) to balance empirical loss with data-driven variance, extended to off-policy learning (ASVP-PL) and online updates (OSVP-L). Theoretical results provide variance-adaptive excess-risk and regret bounds under bounded-weight and margin assumptions, with rates that interpolate between parametric and nonparametric regimes via the sequential entropy exponent $p$ and the margin parameter $\beta$. Empirical studies on dependent data and both continuous and discrete action spaces confirm improved stability and performance over traditional estimators, particularly under limited exploration and suboptimal logging. The framework unifies Bernstein-type variance penalization with adaptive, sequential policy updates, offering practical impact for online systems, offline evaluation, and beyond.

Abstract

Adaptive experiments produce dependent data that break i.i.d. assumptions that underlie classical concentration bounds and invalidate standard learning guarantees. In this paper, we develop a self-normalized maximal inequality for martingale empirical processes. Building on this, we first propose an adaptive sample-variance penalization procedure which balances empirical loss and sample variance, valid for general dependent data. Next, this allows us to derive a new variance-regularized pessimistic off-policy learning objective, for which we establish excess-risk guarantees. Subsequently, we show that, when combined with sequential updates and under standard complexity and margin conditions, the resulting estimator achieves fast convergence rates in both parametric and nonparametric regimes, improving over the usual $1/\sqrt{n}$ baseline. We complement our theoretical findings with numerical simulations that illustrate the practical gains of our approach.

Online Policy Learning via a Self-Normalized Maximal Inequality

TL;DR

The paper tackles policy learning with adaptively collected data, where dependence undermines classical i.i.d. guarantees. It develops a self-normalized maximal inequality for martingale empirical processes and builds Adaptive Sample Variance Penalization (ASVP) to balance empirical loss with data-driven variance, extended to off-policy learning (ASVP-PL) and online updates (OSVP-L). Theoretical results provide variance-adaptive excess-risk and regret bounds under bounded-weight and margin assumptions, with rates that interpolate between parametric and nonparametric regimes via the sequential entropy exponent and the margin parameter . Empirical studies on dependent data and both continuous and discrete action spaces confirm improved stability and performance over traditional estimators, particularly under limited exploration and suboptimal logging. The framework unifies Bernstein-type variance penalization with adaptive, sequential policy updates, offering practical impact for online systems, offline evaluation, and beyond.

Abstract

Adaptive experiments produce dependent data that break i.i.d. assumptions that underlie classical concentration bounds and invalidate standard learning guarantees. In this paper, we develop a self-normalized maximal inequality for martingale empirical processes. Building on this, we first propose an adaptive sample-variance penalization procedure which balances empirical loss and sample variance, valid for general dependent data. Next, this allows us to derive a new variance-regularized pessimistic off-policy learning objective, for which we establish excess-risk guarantees. Subsequently, we show that, when combined with sequential updates and under standard complexity and margin conditions, the resulting estimator achieves fast convergence rates in both parametric and nonparametric regimes, improving over the usual baseline. We complement our theoretical findings with numerical simulations that illustrate the practical gains of our approach.
Paper Structure (63 sections, 11 theorems, 144 equations, 5 figures, 1 table)

This paper contains 63 sections, 11 theorems, 144 equations, 5 figures, 1 table.

Table of Contents

  1. INTRODUCTION
  2. Martingale Framework and Notation
  3. RELATED WORK
  4. SELF-NORMALIZED MAXIMAL INEQUALITY AND VARIANCE PENALIZED LEARNING
  5. Self-Normalized Maximal Inequality
  6. Adaptive Sample Variance Penalization
  7. POLICY LEARNING FROM ADAPTIVELY COLLECTED DATA
  8. Policy Evaluation and Learning Setting
  9. Variance-Penalized Objective for Off-Policy Learning
  10. Excess Risk Analysis
  11. ONLINE SVP POLICY LEARNING
  12. Discussion.
  13. NUMERICAL EXPERIMENTS
  14. ASVP versus ERM on dependent data
  15. Practical benefits of ASVP-PL
  16. ...and 48 more sections

Key Result

Theorem 3.3

Under Assumptions assum:bounded_empirical_process, entropy_assumption, For any $\delta\in(0,1)$, with prob. $\ge 1-\delta$, for all $\ell(f)\in\mathcal{L}_T(\mathcal{F})$, either $\sigma_T(f)\lesssim T^{-1/(2+p)}$, or

Figures (5)

  • Figure 1: Loss evolution and optimization paths for (a) CRM, (b) SCRM, and (c) OSVP-L in example \ref{['ex:gaussian-quadratic']}
  • Figure 2: ERM vs. ASVP in dependent setting.
  • Figure 3: Practical benefits of ASVP-PL under outcome stochasticity and suboptimal logging.
  • Figure 4: OSVP-PL remains robust to poor initial policies and low exploration, outperforming CRM and SCRM.
  • Figure 5: Comparison of ERM and SVP performance.

Theorems & Definitions (22)

  • Theorem 3.3: Self-normalized maximal inequality
  • Remark 3.4
  • Theorem 3.5: ASVP Excess Risk
  • Proposition 4.2: ASVP-PL Excess Risk Bound
  • Example 5.1: Gaussian policies with quadratic loss zenati23scrm
  • Theorem 5.4: OSVP-PL Regret Bound
  • Theorem B.1: Theorem bound; cf. van_Handel_2010
  • Lemma B.2: Localized convergence by peeling
  • proof
  • Lemma B.3: Lemma 2.1 of vandervaart2010localmaximalinequalityuniform
  • ...and 12 more