ART: Adaptive Resampling-based Training for Imbalanced Classification

TL;DR

ART addresses the challenge of class imbalance by introducing a metric-driven, class-level adaptive resampling mechanism. It replaces static class priors with a time-varying distribution $p_i^{(t)}$, updated at intervals determined by the boost frequency $bf$, and blended with empirical priors via $p_i = c\,\Pi_i + (1-c)\,w_i$, where adaptive weights $w_i$ derive from difficulty scores $s_i = 1 - f_i$ based on per-class validation $F1$ scores. This dynamic ERM framework concentrates training on underperforming classes, yielding statistically significant macro $F1$ improvements across tabular, image, and text datasets, and demonstrating robustness to hyperparameter settings and reduced model widths. The approach is modality-agnostic and practical to integrate, offering a reliable alternative to static resampling, loss reweighting, or instance-level difficulty estimation in imbalanced classification.

Abstract

Traditional resampling methods for handling class imbalance typically uses fixed distributions, undersampling the majority or oversampling the minority. These static strategies ignore changes in class-wise learning difficulty, which can limit the overall performance of the model. This paper proposes an Adaptive Resampling-based Training (ART) method that periodically updates the distribution of the training data based on the class-wise performance of the model. Specifically, ART uses class-wise macro F1 scores, computed at fixed intervals, to determine the degree of resampling to be performed. Unlike instance-level difficulty modeling, which is noisy and outlier-sensitive, ART adapts at the class level. This allows the model to incrementally shift its attention towards underperforming classes in a way that better aligns with the optimization objective. Results on diverse benchmarks, including Pima Indians Diabetes and Yeast dataset demonstrate that ART consistently outperforms both resampling-based and algorithm-level methods, including Synthetic Minority Oversampling Technique (SMOTE), NearMiss Undersampling, and Cost-sensitive Learning on binary as well as multi-class classification tasks with varying degrees of imbalance. In most settings, these improvements are statistically significant. On tabular datasets, gains are significant under paired t-tests and Wilcoxon tests (p < 0.05), while results on text and image tasks remain favorable. Compared to training on the original imbalanced data, ART improves macro F1 by an average of 2.64 percentage points across all tested tabular datasets. Unlike existing methods, whose performance varies by task, ART consistently delivers the strongest macro F1, making it a reliable choice for imbalanced classification.

Figures (7)

• Figure 1: ART resampling
• Figure 2: Overview of the training procedure in ART. The process begins with uniform initialization of class weights. Training proceeds in epochs, with periodic updates to class weights every bf epochs based on per-class macro F1-scores evaluated on a validation set.
• Figure 3: Delta-Matrix heatmap showing the difference in macro F1 scores between ART and competing methods across all datasets. Positive (green) values indicate improvements of ART over the compared method, while negative (red) values denote cases where the competing method performs better.
• Figure 4: Comparison of ART performance over different blending constant ($c$) where c is a hyperparameter. Results show ART is insensitive to the choice of $c$ and requires minimal hyperparameter optimization.
• Figure 5: Comparison of performance of ART across varying boosting frequency ($bf$) values while keeping all other hyperparameters constant. Experiments over varying c values show performance is insensitive to choice of boosting frequency.