Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Smoothed Online Classification can be Harder than Batch Classification

Vinod Raman, Unique Subedi, Ambuj Tewari

TL;DR

The paper investigates online classification under smoothed adversaries and shows that smoothed online learnability can diverge from batch PAC learnability when the label space $|\mathcal{Y}|$ is unbounded. It proves a qualitative separation by constructing a PAC-learnable class that is not learnable in the smoothed online model under a uniform base measure, and it provides a quantitative lower bound on regret that grows linearly with $T$ in certain regimes. To reconcile these results, the authors introduce UBEME, a property ensuring smoothed online learnability, and show that UBEME implies bounded empirical metric entropy with respect to the base measure; they also derive a corollary using Graph dimension for finite label spaces. Collectively, the work demonstrates that PAC learnability is not in general sufficient for smoothed online learnability in the unbounded-label setting and highlights the need for joint $(\mathcal{H}, \mu)$-dependent complexity measures to characterize learnability in this regime.

Abstract

We study online classification under smoothed adversaries. In this setting, at each time point, the adversary draws an example from a distribution that has a bounded density with respect to a fixed base measure, which is known apriori to the learner. For binary classification and scalar-valued regression, previous works \citep{haghtalab2020smoothed, block2022smoothed} have shown that smoothed online learning is as easy as learning in the iid batch setting under PAC model. However, we show that smoothed online classification can be harder than the iid batch classification when the label space is unbounded. In particular, we construct a hypothesis class that is learnable in the iid batch setting under the PAC model but is not learnable under the smoothed online model. Finally, we identify a condition that ensures that the PAC learnability of a hypothesis class is sufficient for its smoothed online learnability.

Smoothed Online Classification can be Harder than Batch Classification

TL;DR

The paper investigates online classification under smoothed adversaries and shows that smoothed online learnability can diverge from batch PAC learnability when the label space is unbounded. It proves a qualitative separation by constructing a PAC-learnable class that is not learnable in the smoothed online model under a uniform base measure, and it provides a quantitative lower bound on regret that grows linearly with in certain regimes. To reconcile these results, the authors introduce UBEME, a property ensuring smoothed online learnability, and show that UBEME implies bounded empirical metric entropy with respect to the base measure; they also derive a corollary using Graph dimension for finite label spaces. Collectively, the work demonstrates that PAC learnability is not in general sufficient for smoothed online learnability in the unbounded-label setting and highlights the need for joint -dependent complexity measures to characterize learnability in this regime.

Abstract

We study online classification under smoothed adversaries. In this setting, at each time point, the adversary draws an example from a distribution that has a bounded density with respect to a fixed base measure, which is known apriori to the learner. For binary classification and scalar-valued regression, previous works \citep{haghtalab2020smoothed, block2022smoothed} have shown that smoothed online learning is as easy as learning in the iid batch setting under PAC model. However, we show that smoothed online classification can be harder than the iid batch classification when the label space is unbounded. In particular, we construct a hypothesis class that is learnable in the iid batch setting under the PAC model but is not learnable under the smoothed online model. Finally, we identify a condition that ensures that the PAC learnability of a hypothesis class is sufficient for its smoothed online learnability.
Paper Structure (13 sections, 14 theorems, 56 equations)

This paper contains 13 sections, 14 theorems, 56 equations.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Smoothed Online Learning
  4. PAC Learning and Sample Compression Schemes
  5. Covering Numbers, Metric Entropy, and Complexity Measures
  6. PAC Learnability is Not Sufficient for Smoothed Online Learnability
  7. A Sufficient Condition for Smoothed Online Classification
  8. Discussion
  9. PAC Learnability, Combinatorial Dimensions, Complexity Measures
  10. Helper Lemmas
  11. Proof of Theorem \ref{['thm:quant_sep']}
  12. Proof of Theorem \ref{['thm:suff']}
  13. Proof of Corollary \ref{['cor:graph']}

Key Result

Theorem 1

(Informal) Let $\mathcal{X} = [0,1]$. Then, there exists $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$ that is PAC learnable but not learnable in the smoothed online setting with $\mu = \text{Uniform}(\mathcal{X})$ and $\sigma =1$.

Theorems & Definitions (30)

  • Theorem
  • Theorem
  • Definition 1: Smoothed Online Learnability
  • Theorem 2.1: Compression $\implies$ Learnability david2016statistical
  • Definition 2: Covering Number
  • Definition 3: Uniformly Bounded Empirical Metric Entropy
  • Theorem 3.1
  • Corollary 3.2: Agnostic PAC Learnability $\nRightarrow$ Smoothed Online Learnability
  • proof
  • Theorem 3.3
  • ...and 20 more