Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Some Robustness Properties of Label Cleaning

Chen Cheng, John Duchi

TL;DR

It is demonstrated that learning procedures that rely on aggregated labels, e.g., label information distilled from noisy responses, enjoy robustness properties impossible without data cleaning, highlighting how incorporating a fuller view of the data analysis pipeline can yield a more robust methodology by refining noisy signals.

Abstract

We demonstrate that learning procedures that rely on aggregated labels, e.g., label information distilled from noisy responses, enjoy robustness properties impossible without data cleaning. This robustness appears in several ways. In the context of risk consistency -- when one takes the standard approach in machine learning of minimizing a surrogate (typically convex) loss in place of a desired task loss (such as the zero-one mis-classification error) -- procedures using label aggregation obtain stronger consistency guarantees than those even possible using raw labels. And while classical statistical scenarios of fitting perfectly-specified models suggest that incorporating all possible information -- modeling uncertainty in labels -- is statistically efficient, consistency fails for ``standard'' approaches as soon as a loss to be minimized is even slightly mis-specified. Yet procedures leveraging aggregated information still converge to optimal classifiers, highlighting how incorporating a fuller view of the data analysis pipeline, from collection to model-fitting to prediction time, can yield a more robust methodology by refining noisy signals.

Some Robustness Properties of Label Cleaning

TL;DR

It is demonstrated that learning procedures that rely on aggregated labels, e.g., label information distilled from noisy responses, enjoy robustness properties impossible without data cleaning, highlighting how incorporating a fuller view of the data analysis pipeline can yield a more robust methodology by refining noisy signals.

Abstract

We demonstrate that learning procedures that rely on aggregated labels, e.g., label information distilled from noisy responses, enjoy robustness properties impossible without data cleaning. This robustness appears in several ways. In the context of risk consistency -- when one takes the standard approach in machine learning of minimizing a surrogate (typically convex) loss in place of a desired task loss (such as the zero-one mis-classification error) -- procedures using label aggregation obtain stronger consistency guarantees than those even possible using raw labels. And while classical statistical scenarios of fitting perfectly-specified models suggest that incorporating all possible information -- modeling uncertainty in labels -- is statistically efficient, consistency fails for ``standard'' approaches as soon as a loss to be minimized is even slightly mis-specified. Yet procedures leveraging aggregated information still converge to optimal classifiers, highlighting how incorporating a fuller view of the data analysis pipeline, from collection to model-fitting to prediction time, can yield a more robust methodology by refining noisy signals.

Paper Structure

This paper contains 54 sections, 25 theorems, 205 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Label aggregation
  4. Surrogate consistency
  5. Fisher consistency failure without label aggregation: ranking
  6. Label aggregation obtains stronger surrogate consistency
  7. Basic extensions of surrogate consistency
  8. Identifying surrogates and consistency
  9. Identifying surrogates and consistency amplification
  10. Surrogate consistency examples with majority vote
  11. Binary classification with a nonsmooth surrogate
  12. Bipartite matching
  13. Structured prediction
  14. Robustness and consistency for models
  15. Consistency failure for binary classification in finite dimensions
  16. ...and 39 more sections

Key Result

Corollary 3.1

The surrogate $\varphi$ is Fisher consistent item:pointwise-comparison for $\ell$ if and only if $\psi(\epsilon, x) > 0$ for all $x \in \mathcal{X}$ and $\epsilon > 0$. Let $\psi$ be the Fenchel biconjugate of $\overline{\psi}$. Then $\overline{\psi}(\epsilon) > 0$ if and only if $\psi(\epsilon) > 0

Figures (2)

  • Figure 1: Numerical illustrations for truncated quadratic surrogate $\phi(\delta) = \max\{1-\delta, 0\}^2$, showing comparison inequalities for non-aggregated data $\psi$ (optimal) vs. majority vote aggregation $\psi_m$ (suboptimal) when $m \in \{2^{16}, 2^{20}, 2^{24}, 2^{28}, 2^{32}\}$.
  • Figure 2: Numerical illustrations for mis-specified logistic models when $d=2$ and $k=3$. The true parameters are $\theta_1^\star = (0, 1)^\top, \theta_2^\star = (1, 1)^\top$. The corrupted link parameters are $r = 3$ and $\alpha = 0.1$. The plot shows the error of classification rule $\left\|\|T_m\| - T^\star / \|T^\star\|\right\|$ vs. number of labels $m \in \{1, 4, 16, 64, 256, 1024\}$. For each $m$, the synthetic dataset consist of $n=10,000$ multilabeled data points $(X_i, (Y_{i1}, \cdots, Y_{im}))$ from $\tilde{\sigma}$. We report a $95\%$ error bar on $T=100$ trials for each $n$.

Theorems & Definitions (27)

  • Corollary 3.1
  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Definition 3.1: Identifying surrogate
  • Corollary 3.2
  • Definition 3.2
  • Theorem 1
  • Corollary 3.3
  • Lemma 3.1
  • ...and 17 more