Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Minimax Boosting with Performance Guarantees

Santiago Mazuelas, Veronica Alvarez

TL;DR

RMBoost reframes boosting as a minimax robust learning problem, minimizing the worst-case misclassification probability under a noise-aware uncertainty set defined by a family of base-rules. The authors derive a convex dual that reduces to an L1-regularized linear program, enabling sparse, margin-aware classifiers without selecting a fixed potential function. They provide finite-sample generalization bounds that quantify the impact of label noise, sample size, and proximity to Bayes risk, and establish Bayes-consistency under common approximation assumptions. An efficient sequential learning algorithm based on column generation is developed to construct RMBoost models, and extensive experiments show strong accuracy on clean data and enhanced resilience to symmetric, uniform, and adversarial label noise, often outperforming existing robust boosting methods.

Abstract

Boosting methods often achieve excellent classification accuracy, but can experience notable performance degradation in the presence of label noise. Existing robust methods for boosting provide theoretical robustness guarantees for certain types of label noise, and can exhibit only moderate performance degradation. However, previous theoretical results do not account for realistic types of noise and finite training sizes, and existing robust methods can provide unsatisfactory accuracies, even without noise. This paper presents methods for robust minimax boosting (RMBoost) that minimize worst-case error probabilities and are robust to general types of label noise. In addition, we provide finite-sample performance guarantees for RMBoost with respect to the error obtained without noise and with respect to the best possible error (Bayes risk). The experimental results corroborate that RMBoost is not only resilient to label noise but can also provide strong classification accuracy.

Robust Minimax Boosting with Performance Guarantees

TL;DR

RMBoost reframes boosting as a minimax robust learning problem, minimizing the worst-case misclassification probability under a noise-aware uncertainty set defined by a family of base-rules. The authors derive a convex dual that reduces to an L1-regularized linear program, enabling sparse, margin-aware classifiers without selecting a fixed potential function. They provide finite-sample generalization bounds that quantify the impact of label noise, sample size, and proximity to Bayes risk, and establish Bayes-consistency under common approximation assumptions. An efficient sequential learning algorithm based on column generation is developed to construct RMBoost models, and extensive experiments show strong accuracy on clean data and enhanced resilience to symmetric, uniform, and adversarial label noise, often outperforming existing robust boosting methods.

Abstract

Boosting methods often achieve excellent classification accuracy, but can experience notable performance degradation in the presence of label noise. Existing robust methods for boosting provide theoretical robustness guarantees for certain types of label noise, and can exhibit only moderate performance degradation. However, previous theoretical results do not account for realistic types of noise and finite training sizes, and existing robust methods can provide unsatisfactory accuracies, even without noise. This paper presents methods for robust minimax boosting (RMBoost) that minimize worst-case error probabilities and are robust to general types of label noise. In addition, we provide finite-sample performance guarantees for RMBoost with respect to the error obtained without noise and with respect to the best possible error (Bayes risk). The experimental results corroborate that RMBoost is not only resilient to label noise but can also provide strong classification accuracy.
Paper Structure (25 sections, 8 theorems, 85 equations, 7 figures, 5 tables, 1 algorithm)

This paper contains 25 sections, 8 theorems, 85 equations, 7 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem formulation
  4. Related work
  5. Minimax boosting
  6. Generalization and robustness guarantees
  7. Efficient sequential learning for RMBoost
  8. Learning algorithm
  9. Theoretical analysis
  10. Numerical results
  11. Conclusion
  12. Limitations:
  13. Strong duality lemma
  14. Auxiliary Lemmas
  15. Proof of Theorem \ref{['th1']}
  16. ...and 10 more sections

Key Result

Theorem 1

If $\boldsymbol{\mu} ^*$ is a solution of dual, the classification rule $\mathrm{h}_{ \boldsymbol{\mu} ^*}\in\text{T}(\mathcal{X},\mathcal{Y})$ given by is a solution of the minimax problem in minimax. In addition, the minimax risk $\overline{R}$ coincides with the optimum of dual, that is $\overline{R}=F( \boldsymbol{\mu} ^*)$.

Figures (7)

  • Figure 1: Trade-off classification performance vs robustness to noise in the 8 datasets (uniform and symmetric noise with $P_{\text{noise}}=10\%$).
  • Figure 2: Performance degradation of AdaBoost and RMBoost methods for increased levels of noise in 'Credit' dataset.
  • Figure 3: Trade-off classification performance vs robustness to uniform and symmetric noise as well as adversarial noise.
  • Figure 4: Performance degradation of AdaBoost and RMBoost methods for increased levels of noise.
  • Figure 5: Comparison of relative running times vs training sizes for RMBoost, LPBoost, and AdaBoost.
  • ...and 2 more figures

Theorems & Definitions (16)

  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Theorem 3
  • proof
  • Theorem 4
  • proof
  • Lemma 5
  • proof
  • ...and 6 more