MoMBS: Mixed-order minibatch sampling enhances model training from diverse-quality images

TL;DR

MoMBS tackles learning from diverse-quality images by jointly using loss and uncertainty to assess sample difficulty and by mixing samples of varying difficulty within minibatches. The method introduces an assessor to compute a robust difficulty score and a scheduler that constructs minibatches to maximize positive updates while limiting harmful influence from poorly labeled or overfitted samples. Across universal lesion detection, COVID-19 CT segmentation, long-tailed, and noisy-label CIFAR-100 tasks, MoMBS yields consistent improvements, particularly with limited training data, while remaining architecture-agnostic. This approach enhances training robustness in realistic, quality-variant datasets and can simplify deployment by reducing reliance on specialized network designs.

Abstract

Natural images exhibit label diversity (clean vs. noisy) in noisy-labeled image classification and prevalence diversity (abundant vs. sparse) in long-tailed image classification. Similarly, medical images in universal lesion detection (ULD) exhibit substantial variations in image quality, encompassing attributes such as clarity and label correctness. How to effectively leverage training images with diverse qualities becomes a problem in learning deep models. Conventional training mechanisms, such as self-paced curriculum learning (SCL) and online hard example mining (OHEM), relieve this problem by reweighting images with high loss values. Despite their success, these methods still confront two challenges: (i) the loss-based measure of sample hardness is imprecise, preventing optimum handling of different cases, and (ii) there exists under-utilization in SCL or over-utilization OHEM with the identified hard samples. To address these issues, this paper revisits the minibatch sampling (MBS), a technique widely used in deep network training but largely unexplored concerning the handling of diverse-quality training samples. We discover that the samples within a minibatch influence each other during training; thus, we propose a novel Mixed-order Minibatch Sampling (MoMBS) method to optimize the use of training samples with diverse qualities. MoMBS introduces a measure that takes both loss and uncertainty into account to surpass a sole reliance on loss and allows for a more refined categorization of high-loss samples by distinguishing them as either poorly labeled and under represented or well represented and overfitted. We prioritize under represented samples as the main gradient contributors in a minibatch and keep them from the negative influences of poorly labeled or overfitted samples with a mixed-order minibatch sampling design.

Figures (7)

• Figure 1: Left: Training samples are typically grouped into four types based on data loss and uncertainty: ($s_p$) poorly labeled, ($s_u$) under represented, ($s_w$) well represented and ($s_o$) overfitted samples. The loss-based sample quality measurer, employed in self-paced curriculum learning (SCL) and online hard example mining (OHEM), inaccurately treats both ($s_p$) and ($s_u$) as low-quality samples. Right: The distribution comparison of different minibatches obtained with random vs. the proposed mixed-order minibatch sampling (MoMBS). The MoMBS first categorizes a minibatch into positive and negative based on the types of training samples it contains and then, through mixed-order sampling, it increases the number of positive minibatchs and decreases the number of negative minibatchs, thereby enhancing deep network's parameters updating. In comparison, random MBS creates a large number of negative minibatchs. Note that the above plots and statistics are derived from the universal lesion detection experiment.
• Figure 2: Our MoMBS method consists of three learning steps. In step A, we randomly sample data using the vanilla random sampler for initial network training. After $e$ epochs, we activate the uncertainty estimation component (Eq. \ref{['Equ1']}) to calculate the uncertainty of each sample. In step B, we sort all samples based on their loss and uncertainty ranks and calculate the sum of the rank indices of uncertainty $\pi(i)$ and rank indices of loss $\tau(i)$ to obtain their difficulty rank score $d$. Finally, in step C, we rank all training samples based on their difficulty rank score $d$ and construct minibatches by pairing samples with high $d$ with those with low $d$. The newly formed minibatches are used for the subsequent network training.
• Figure 3: ($a_{1}$): Total loss reduction after one iteration backward for two samples in a minibatch vs. their individual loss values. ($b_{1}$): Loss reduction after one iteration backward of the sample with the lower loss in a minibatch vs. their individual loss values. ($a_{2}$): Total loss reduction after one iteration backward vs. the sum of their individual loss and uncertainty values. ($b_{2}$): Loss reduction after one iteration backward of the sample with the lower loss in a minibatch vs. the sum of their individual loss and uncertainty values.
• Figure 4: Ablation study for Batchsize (BS) and pivot epoch based on DeepLesion cai2020lesion, CIFAR100-LT zhou2020bbn, CIFAR100N with human noise wei2021learning and CIFAR100N with symmetric noise ma2020normalized.
• Figure 6: Illustration of loss and uncertainty relationship based on li2022satr. Yellow (or cyan) denotes the sample whose absolute difference between uncertainty and loss is greater (or less) than 0.3.