Automatic Combination of Sample Selection Strategies for Few-Shot Learning

TL;DR

The paper tackles the sensitivity of few-shot learning to sample selection by exhaustively evaluating 20 selection strategies across multiple few-shot approaches and domains. It introduces ACSESS, an automatic method that identifies a subset of complementary strategies and combines them to score and select samples, achieving consistent improvements over single strategies and a recent in-context learning baseline. Benefits are strongest at low shot counts and diminish as the number of shots increases, with learnability emerging as a key determinant of sample quality. The work provides practical guidance on when to apply strategy combinations and emphasizes the importance of leveraging diverse, complementary properties to boost performance in data-scarce regimes.

Abstract

In few-shot learning, such as meta-learning, few-shot fine-tuning or in-context learning, the limited number of samples used to train a model have a significant impact on the overall success. Although a large number of sample selection strategies exist, their impact on the performance of few-shot learning is not extensively known, as most of them have been so far evaluated in typical supervised settings only. In this paper, we thoroughly investigate the impact of 20 sample selection strategies on the performance of 5 few-shot learning approaches over 8 image and 6 text datasets. In addition, we propose a new method for automatic combination of sample selection strategies (ACSESS) that leverages the strengths and complementary information of the individual strategies. The experimental results show that our method consistently outperforms the individual selection strategies, as well as the recently proposed method for selecting support examples for in-context learning. We also show a strong modality, dataset and approach dependence for the majority of strategies as well as their dependence on the number of shots - demonstrating that the sample selection strategies play a significant role for lower number of shots, but regresses to random selection at higher number of shots.

Figures (19)

• Figure 1: Benefit of the different selection strategies, calculated as the difference in accuracy between the specific strategy and the classic few-shot selection, aggregated over the image and text datasets (boxplots show the distribution of results across the datasets). The performance of the classic selection is represented as the red dashed line (zero value). The consistently beneficial selection strategies depend on the data modality (image vs. text) and the approach (gradient few-shot learning using meta-learning and fine-tuning vs. in-context learning using large language models). Our proposed method ACSESS consistently leads to improved performance.
• Figure 2: Comparison of the LENS method li-qiu-2023-finding to our proposed automatic combination of selection strategies (ACSESS).
• Figure 3: Effect of the number of shots on the benefit of selection strategies, comparing ACSESS and random selection, aggregated over datasets. The benefit is calculated as a difference to the classic selection at 5-shots. The benefit of sample selection is more significant at lower number of shots. At larger number of shots, the benefit and the performance boost from further increase of shots becomes negligible.
• Figure 4: Standard deviation introduced by multiple runs of the different selection strategies. The results for Prototypical Networks, MAML and Few-Shot Fine-Tuning are aggregated over both the image and text datasets, while the results for Mistral and Zephyr are only from text datasets. The ACSESS method shows lower sensitivity to repeated runs compared to the majority of the strategies included in the combination.
• Figure 5: Distribution of the selection strategies benefit (calculated as difference to the classic few-shot selection) for the 5-shot and 10-shot setting. The benefit of the different selection strategies is more significant at lower number of shots.