Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Testing Noise Assumptions of Learning Algorithms

Surbhi Goel, Adam R. Klivans, Konstantinos Stavropoulos, Arsen Vasilyan

TL;DR

The paper tackles whether one can efficiently verify the noise-model assumptions of a training dataset within the framework of testable learning. It provides a polynomial-time tester-learner for origin-centered halfspaces with Gaussian marginals under Massart noise, using disagreement and spectral testers and sandwiching techniques to certify near-optimality when accepted. A key finding is a sharp separation: for random classification noise at η = 1/2, testable learning is hard while classical learning is trivial, highlighting a distinct boundary between the two paradigms as noise tightens. These results inform algorithm selection under uncertain data-noise conditions and demonstrate the practical impact of verifying noise-model assumptions prior to learning.

Abstract

We pose a fundamental question in computational learning theory: can we efficiently test whether a training set satisfies the assumptions of a given noise model? This question has remained unaddressed despite decades of research on learning in the presence of noise. In this work, we show that this task is tractable and present the first efficient algorithm to test various noise assumptions on the training data. To model this question, we extend the recently proposed testable learning framework of Rubinfeld and Vasilyan (2023) and require a learner to run an associated test that satisfies the following two conditions: (1) whenever the test accepts, the learner outputs a classifier along with a certificate of optimality, and (2) the test must pass for any dataset drawn according to a specified modeling assumption on both the marginal distribution and the noise model. We then consider the problem of learning halfspaces over Gaussian marginals with Massart noise (where each label can be flipped with probability less than $1/2$ depending on the input features), and give a fully-polynomial time testable learning algorithm. We also show a separation between the classical setting of learning in the presence of structured noise and testable learning. In fact, for the simple case of random classification noise (where each label is flipped with fixed probability $η= 1/2$), we show that testable learning requires super-polynomial time while classical learning is trivial.

Testing Noise Assumptions of Learning Algorithms

TL;DR

The paper tackles whether one can efficiently verify the noise-model assumptions of a training dataset within the framework of testable learning. It provides a polynomial-time tester-learner for origin-centered halfspaces with Gaussian marginals under Massart noise, using disagreement and spectral testers and sandwiching techniques to certify near-optimality when accepted. A key finding is a sharp separation: for random classification noise at η = 1/2, testable learning is hard while classical learning is trivial, highlighting a distinct boundary between the two paradigms as noise tightens. These results inform algorithm selection under uncertain data-noise conditions and demonstrate the practical impact of verifying noise-model assumptions prior to learning.

Abstract

We pose a fundamental question in computational learning theory: can we efficiently test whether a training set satisfies the assumptions of a given noise model? This question has remained unaddressed despite decades of research on learning in the presence of noise. In this work, we show that this task is tractable and present the first efficient algorithm to test various noise assumptions on the training data. To model this question, we extend the recently proposed testable learning framework of Rubinfeld and Vasilyan (2023) and require a learner to run an associated test that satisfies the following two conditions: (1) whenever the test accepts, the learner outputs a classifier along with a certificate of optimality, and (2) the test must pass for any dataset drawn according to a specified modeling assumption on both the marginal distribution and the noise model. We then consider the problem of learning halfspaces over Gaussian marginals with Massart noise (where each label can be flipped with probability less than depending on the input features), and give a fully-polynomial time testable learning algorithm. We also show a separation between the classical setting of learning in the presence of structured noise and testable learning. In fact, for the simple case of random classification noise (where each label is flipped with fixed probability ), we show that testable learning requires super-polynomial time while classical learning is trivial.
Paper Structure (26 sections, 17 theorems, 116 equations, 2 figures, 1 table, 2 algorithms)

This paper contains 26 sections, 17 theorems, 116 equations, 2 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Our Results
  3. Our Techniques
  4. Related Work
  5. Polynomial-Time Tester-Learners
  6. Warm-up: Random Classification Noise Oracles
  7. Massart Noise Oracles
  8. Lower Bounds in the High-Noise Regime
  9. Cryptographic Hardness
  10. SQ Lower Bounds
  11. Preliminaries
  12. Some standard notation.
  13. Standard lemmas.
  14. Disagreement Tester
  15. Completeness
  16. ...and 11 more sections

Key Result

Theorem 2.2

Let $c\in(0,1/2)$ be any constant and $\eta_0 = 1/2 - c$. Then, there is an algorithm that testably learns the class $\mathcal{H}_{\mathsf{hs}}$ with respect to $\mathsf{EX}^{\mathsf{RCN}}_{\mathcal{N}, \mathcal{H}_{\mathsf{hs}}, \eta_0} = \{\mathsf{EX}^{\mathsf{RCN}}_{\mathcal{N}, f, \eta_0}: f\in

Figures (2)

  • Figure 1: The shaded region is $\{\mathbf{x}\in\mathbb{R}^d: \mathop{\mathrm{\mathsf{sign}}}\nolimits(\mathbf{v}\cdot \mathbf{x}) \neq \mathop{\mathrm{\mathsf{sign}}}\nolimits(\mathbf{v}^*\cdot\mathbf{x})\}$. Left: red square points have label $+1$, blue round points have label $-1$. Right: green square points are in $\bar{S}_g$ and purple round points are in $\bar{S}_b$.
  • Figure 2: For vectors $\mathbf{v},\mathbf{v}'\in\mathbb{S}^{d-1}$, the region $\{\mathbf{x}\in\mathbb{R}^d: \mathop{\mathrm{\mathsf{sign}}}\nolimits(\mathbf{v}\cdot \mathbf{x}) \neq \mathop{\mathrm{\mathsf{sign}}}\nolimits(\mathbf{v}'\cdot\mathbf{x})\}$ is contained in the union of green regions and it contains the union of blue regions. In the diagram we highlight one of the green regions (top left) and one of the blue regions (bottom right).

Theorems & Definitions (50)

  • Definition 1.1: Massart Noise Oracle
  • Definition 1.2: Origin-Centered Halfspaces
  • Definition 1.3: Testable Learning, extension of Definition 4 in rubinfeld2022testing
  • Definition 2.1: Random Classification Noise (RCN) Oracle
  • Theorem 2.2: Warm-up: RCN
  • Theorem 2.4: Disagreement tester, see \ref{['thm: disagreement tester']}
  • Lemma 2.5: Informal
  • Theorem 2.7: Main Result
  • Theorem 2.9: Spectral tester, see \ref{['thm: spectral tester']}
  • proof
  • ...and 40 more