Learning with Monotone Adversarial Corruptions
Kasper Green Larsen, Chirag Pabbaraju, Abhishek Shetty
TL;DR
The paper studies how relaxing exchangeability via a monotone adversary affects generalization in binary classification. It shows that even optimal leave-one-out and ensemble methods can be forced to fail (constant test error), while uniform-convergence-based ERM methods remain robust with rates of $O\left(\frac{d\log(n/d)}{n}\right)$; in oblivious-adversary settings, the One-inclusion Graph achieves $O\left(\frac{d}{n}\right)$. The work provides tight lower bounds for majority-voting approaches and OIG under monotone corruption, underscoring the fragility of exchangeability-based guarantees and the resilience of ERM-based strategies. These results illuminate how small violations of independence can drastically alter learnability and guide future study on robust learning under dependent data.
Abstract
We study the extent to which standard machine learning algorithms rely on exchangeability and independence of data by introducing a monotone adversarial corruption model. In this model, an adversary, upon looking at a "clean" i.i.d. dataset, inserts additional "corrupted" points of their choice into the dataset. These added points are constrained to be monotone corruptions, in that they get labeled according to the ground-truth target function. Perhaps surprisingly, we demonstrate that in this setting, all known optimal learning algorithms for binary classification can be made to achieve suboptimal expected error on a new independent test point drawn from the same distribution as the clean dataset. On the other hand, we show that uniform convergence-based algorithms do not degrade in their guarantees. Our results showcase how optimal learning algorithms break down in the face of seemingly helpful monotone corruptions, exposing their overreliance on exchangeability.