Multi-Label Adaptive Batch Selection by Highlighting Hard and Imbalanced Samples

TL;DR

This work tackles the challenge of class imbalance in multi-label learning by introducing an adaptive batch selection strategy that prioritizes hard minority-label samples. The method combines rank-based sampling with imbalance-aware weighting and a quantization mechanism to stabilize batch probabilities, and extends to a label-correlation variant that uses a label adjacency matrix for sequence-based sampling. Empirical results across 13 datasets and five deep models show faster convergence and improved metrics, especially on highly imbalanced or high-cardinality datasets, with convergence guarantees provided. The approach offers practical gains for training efficiency and predictive performance in multi-label domains, and opens avenues for leveraging label dependencies in batch construction.

Abstract

Deep neural network models have demonstrated their effectiveness in classifying multi-label data from various domains. Typically, they employ a training mode that combines mini-batches with optimizers, where each sample is randomly selected with equal probability when constructing mini-batches. However, the intrinsic class imbalance in multi-label data may bias the model towards majority labels, since samples relevant to minority labels may be underrepresented in each mini-batch. Meanwhile, during the training process, we observe that instances associated with minority labels tend to induce greater losses. Existing heuristic batch selection methods, such as priority selection of samples with high contribution to the objective function, i.e., samples with high loss, have been proven to accelerate convergence while reducing the loss and test error in single-label data. However, batch selection methods have not yet been applied and validated in multi-label data. In this study, we introduce a simple yet effective adaptive batch selection algorithm tailored to multi-label deep learning models. It adaptively selects each batch by prioritizing hard samples related to minority labels. A variant of our method also takes informative label correlations into consideration. Comprehensive experiments combining five multi-label deep learning models on thirteen benchmark datasets show that our method converges faster and performs better than random batch selection.

Figures (8)

• Figure 1: The log-scaled loss density distributions of the DELA model trained on the bibtex dataset at the 30-th and 70-th epochs, where blue, yellow, red and green curves denote the loss distribution of instances associated with zero, one, two and more than two minority labels, respectively.
• Figure 2: The work flow of multi-label deep learning model training with adaptive batch selection
• Figure 3: Convergence curves of three batch selection strategies using DELA with Adam optimizer on different datasets.
• Figure 4: Macro-AUC of validation set for batch selection strategies using DELA with Adam optimizer on different datasets.
• Figure 5: The loss distribution of minority samples at 30-th epoch and 70-th epoch under adaptive batch selection methods.