Spuriousness-Aware Meta-Learning for Learning Robust Classifiers
Guangtao Zheng, Wenqian Ye, Aidong Zhang
TL;DR
This work tackles the problem of spurious correlations in image classification without relying on group labels. It proposes SPUME, a framework that first detects spurious attributes using pre-trained vision-language models to produce interpretable text-form attributes, then measures their spuriousness with a metric and constructs spuriousness-aware meta-learning tasks. A centroid-based representation learning approach is trained via meta-learning over these tasks to produce robust features invariant to the detected spurious correlations. Across five benchmarks with diverse biases and distributional shifts, SPUME achieves state-of-the-art robustness, highlighting its practical potential for scalable, annotation-free debiasing in vision systems. The approach is modular and can incorporate stronger VLMs and augments with additional robustness techniques.
Abstract
Spurious correlations are brittle associations between certain attributes of inputs and target variables, such as the correlation between an image background and an object class. Deep image classifiers often leverage them for predictions, leading to poor generalization on the data where the correlations do not hold. Mitigating the impact of spurious correlations is crucial towards robust model generalization, but it often requires annotations of the spurious correlations in data -- a strong assumption in practice. In this paper, we propose a novel learning framework based on meta-learning, termed SPUME -- SPUriousness-aware MEta-learning, to train an image classifier to be robust to spurious correlations. We design the framework to iteratively detect and mitigate the spurious correlations that the classifier excessively relies on for predictions. To achieve this, we first propose to utilize a pre-trained vision-language model to extract text-format attributes from images. These attributes enable us to curate data with various class-attribute correlations, and we formulate a novel metric to measure the degree of these correlations' spuriousness. Then, to mitigate the reliance on spurious correlations, we propose a meta-learning strategy in which the support (training) sets and query (test) sets in tasks are curated with different spurious correlations that have high degrees of spuriousness. By meta-training the classifier on these spuriousness-aware meta-learning tasks, our classifier can learn to be invariant to the spurious correlations. We demonstrate that our method is robust to spurious correlations without knowing them a priori and achieves the best on five benchmark datasets with different robustness measures.