Comparing Importance Sampling Based Methods for Mitigating the Effect of Class Imbalance

TL;DR

This paper tackles the problem of class imbalance in satellite imagery by evaluating three importance-sampling-inspired methods—loss reweighting, undersampling, and oversampling—across two encoders (CLIP and ResNet-18) and two datasets (Planet Rainforest and ADE20K). It finds that up-weighting the loss for underrepresented classes yields negligible improvements, while oversampling generally enhances low-frequency class performance; undersampling can match baselines for low-frequency classes but often harms others. The authors observe potential redundancy in the Planet dataset and demonstrate that results hold across both encoders, suggesting limits to intuitive bias-mitigation strategies for modern vision models. They provide open-source code to foster further work on domain-specific satellite imagery datasets and data-efficient sampling approaches.

Abstract

Most state-of-the-art computer vision models heavily depend on data. However, many datasets exhibit extreme class imbalance which has been shown to negatively impact model performance. Among the training-time and data-generation solutions that have been explored, one subset that leverages existing data is importance sampling. A good deal of this work focuses primarily on the CIFAR-10 and CIFAR-100 datasets which fail to be representative of the scale, composition, and complexity of current state-of-the-art datasets. In this work, we explore and compare three techniques that derive from importance sampling: loss reweighting, undersampling, and oversampling. Specifically, we compare the effect of these techniques on the performance of two encoders on an impactful satellite imagery dataset, Planet's Amazon Rainforest dataset, in preparation for another work. Furthermore, we perform supplemental experimentation on a scene classification dataset, ADE20K, to test on a contrasting domain and clarify our results. Across both types of encoders, we find that up-weighting the loss for and undersampling has a negigible effect on the performance on underrepresented classes. Additionally, our results suggest oversampling generally improves performance for the same underrepresented classes. Interestingly, our findings also indicate that there may exist some redundancy in data in the Planet dataset. Our work aims to provide a foundation for further work on the Planet dataset and similar domain-specific datasets. We open-source our code at https://github.com/RichardZhu123/514-class-imbalance for future work on other satellite imagery datasets as well.

Figures (14)

• Figure 1: Long-tail distribution of classes in the Planet dataset. The frequency of images across the $17$ labels in the Planet dataset form a skewed distribution where some labels, such as "primary" and "agriculture", have many images and other labels, such as "blooming" and "blow_down", have very few images. Important labels, such as "slash_and_burn" and "artisanal_mining", are some of the most underrepresented classes in the dataset.
• Figure 2: Reweighting Loss Method. Our approach scales the loss $\ell(x,y)$ where $x$ is the image and $y$ is the label by a constant factor $k$ if the instance belongs to a low-frequency class. We evaluate with a range of values for $k$ (Section \ref{['sec:results']}), specifically $1.5$, $2$, $2.5$, $3$, $5$, and $10$.
• Figure 3: Types of Resampling. Oversampling (b) trains on repeated elements from underrepresented classes, while undersampling (a) does the opposite, training on a randomly-selected subset of elements from overrepresented classes. The threshold chosen for the number of images to resample or remove is dataset-dependent.
• Figure 4: Distribution of classes in ADE20K.
• Figure 5: Sample Images from the Planet dataset. Figure from Planet's Kaggle website labs_planet_2017. The Planet dataset consists of fine-grain categories such as "conventional mining" and "artisanal mining". Each image has multiple labels one of which is a cloud cover label, such as "partly cloudy" or "clear".