Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Geometric Median (GM) Matching for Robust Data Pruning

Anish Acharya, Inderjit S Dhillon, Sujay Sanghavi

TL;DR

Extensive experiments across several popular deep learning benchmarks indicate that Geometric Median Matching consistently improves over prior state-of-the-art; the gains become more profound at high rates of corruption and aggressive pruning rates; making $\gm$ Matching a strong baseline for future research in robust data pruning.

Abstract

Large-scale data collections in the wild, are invariably noisy. Thus developing data pruning strategies that remain robust even in the presence of corruption is critical in practice. In this work, we propose Geometric Median ($\gm$) Matching -- a herding style greedy algorithm that yields a $k$-subset such that the mean of the subset approximates the geometric median of the (potentially) noisy dataset. Theoretically, we show that $\gm$ Matching enjoys an improved $\gO(1/k)$ scaling over $\gO(1/\sqrt{k})$ scaling of uniform sampling; while achieving {\bf optimal breakdown point} of {\bf 1/2} even under {\bf arbitrary} corruption. Extensive experiments across several popular deep learning benchmarks indicate that $\gm$ Matching consistently improves over prior state-of-the-art; the gains become more profound at high rates of corruption and aggressive pruning rates; making $\gm$ Matching a strong baseline for future research in robust data pruning.

Geometric Median (GM) Matching for Robust Data Pruning

TL;DR

Extensive experiments across several popular deep learning benchmarks indicate that Geometric Median Matching consistently improves over prior state-of-the-art; the gains become more profound at high rates of corruption and aggressive pruning rates; making Matching a strong baseline for future research in robust data pruning.

Abstract

Large-scale data collections in the wild, are invariably noisy. Thus developing data pruning strategies that remain robust even in the presence of corruption is critical in practice. In this work, we propose Geometric Median () Matching -- a herding style greedy algorithm that yields a -subset such that the mean of the subset approximates the geometric median of the (potentially) noisy dataset. Theoretically, we show that Matching enjoys an improved scaling over scaling of uniform sampling; while achieving {\bf optimal breakdown point} of {\bf 1/2} even under {\bf arbitrary} corruption. Extensive experiments across several popular deep learning benchmarks indicate that Matching consistently improves over prior state-of-the-art; the gains become more profound at high rates of corruption and aggressive pruning rates; making Matching a strong baseline for future research in robust data pruning.

Paper Structure

This paper contains 15 sections, 2 theorems, 26 equations, 1 figure, 9 tables, 1 algorithm.

Table of Contents

  1. Background
  2. Geometric Median (GM) Matching
  3. Empirical Evidence
  4. Conclusion
  5. Experiments
  6. Experimental Setup
  7. Ideal (No Corruption) Scenario
  8. Corruption Scenarios
  9. Robustness to Image Corruption
  10. Robustness to Label Corruption
  11. Robustness to Adversarial Attacks
  12. Generalization to Unseen Network / Domain
  13. Complete Proofs
  14. Intermediate Lemmas
  15. Proof of Theorem \ref{['th:gm_match_cvg']}

Key Result

Theorem 1

Suppose that, we are given, a set of $\alpha$-corrupted samples ${\mathcal{D}}$ (def:corruption_model), pretrained proxy model $\phi_{\mathbf{B}}$, and an $\epsilon$ approx. $\textsc{Gm}(\cdot)$ oracle eq:gm. Then, $\textsc{Gm Matching}$ guarantees that the mean of the selected $k$-subset $\bm{\mu}^

Figures (1)

  • Figure 1: Toy Example:0/20/45% of the samples are corrupted i.e. drawn from an adversary chosen distribution (red). We compare several baselines for choosing 10% samples: (Uniform) random sampling, (Easy) selects of samples closest to the centroid. (Hard) Selection of samples farthest from the centroid. (Moderate) selects samples closest to the median distance from the centroid. (Herding) moment matching, (GM Matching) robust moment (GM) matching. Clearly $\textsc{Gm}$ Matching is significantly more robust and diverse than the other approaches even at such high corruption rates.

Theorems & Definitions (6)

  • Definition 1: $\alpha$-corrupted generation process
  • Definition 2: Geometric Median
  • Theorem 1
  • lemma 1
  • proof
  • proof