Few-Shot Learning with Class Imbalance

TL;DR

This analysis compares ten state-of-the-art ML and F SL methods on different imbalance distributions and rebalancing techniques and reveals that some FSL methods display a natural disposition against imbalance while most other approaches produce a performance drop compared to the balanced task without the appropriate mitigation.

Abstract

Few-Shot Learning (FSL) algorithms are commonly trained through Meta-Learning (ML), which exposes models to batches of tasks sampled from a meta-dataset to mimic tasks seen during evaluation. However, the standard training procedures overlook the real-world dynamics where classes commonly occur at different frequencies. While it is generally understood that class imbalance harms the performance of supervised methods, limited research examines the impact of imbalance on the FSL evaluation task. Our analysis compares 10 state-of-the-art meta-learning and FSL methods on different imbalance distributions and rebalancing techniques. Our results reveal that 1) some FSL methods display a natural disposition against imbalance while most other approaches produce a performance drop by up to 17\% compared to the balanced task without the appropriate mitigation; 2) contrary to popular belief, many meta-learning algorithms will not automatically learn to balance from exposure to imbalanced training tasks; 3) classical rebalancing strategies, such as random oversampling, can still be very effective, leading to state-of-the-art performances and should not be overlooked; 4) FSL methods are more robust against meta-dataset imbalance than imbalance at the task-level with a similar imbalance ratio ($ρ<20$), with the effect holding even in long-tail datasets under a larger imbalance ($ρ=65$).

Figures (12)

• Figure 1: The Class Imbalanced Few-Shot Learning (CIFSL) problem investigated in this work. We evaluate imbalance at the task-level (left) in Section \ref{['sec imb task']}, dataset-level (middle) in Section \ref{['sec imb dataset']} and combined (right) in Section \ref{['sec imb combined']}. We study imbalance under several imbalance distributions: $linear$ (task 1), $step$ (task 2), and $random$ (task 3). We also study the effects of $long$-$tail$ imbalance at the meta-dataset level.
• Figure 2: Overview of the Class Imbalance Few-Shot Learning (CIFSL) problem in Standard Meta-Training. Performance of models meta-trained with standard (balanced) tasks, meta-evaluated on balanced and unbalanced distributions. All models show a consistent performance drop when evaluated on imbalance distributions (blue bars) with respect to evaluation on standard balanced tasks (red bar). The task with 1-9shot step imbalance with 1 minority class contains 37 support samples in total, while all other tasks contain only 25 support samples. Most methods perform worse on the imbalanced tasks, despite having the same or higher number of support samples.
• Figure 3: Overview of the Class Imbalance Few-Shot Learning (CIFSL) problem in Random-Shot Meta-Training. Top: Performance of models meta-trained with random-shot tasks, meta-evaluated on balanced and unbalanced distributions. All models show a consistent performance drop when evaluated on imbalance distributions (blue bars) respect to evaluation on standard balanced tasks (red bar). Bottom: Performance difference between Random-Shot Meta-Training Triantafillou2019meta against Standard (balanced) episodic training Vinyals2017matching (i.e. results in Figure \ref{['fig balanced_vs_imbalanced']}) - a positive score indicates an improvement in Random-Shot. The results show that Random-Shot meta-training offers a harder training setting to most algorithms while only some algorithms have better performance on random-shot evaluation tasks.
• Figure 4: Comparing imbalance levels via support sets of different size. Each line represents the average across all models in each training and imbalance setting.
• Figure 5: Average model performance against re-balancing strategies applied at test-time only. Left: all models and training scenarios. Right: performance w.r.t. the balanced task.