SLIM: Spuriousness Mitigation with Minimal Human Annotations

TL;DR

SLIM tackles the problem of spurious correlations undermining model reliability by introducing a human-in-the-loop data construction pipeline that builds an attention-consistent space and curates a feature-balanced subset for training. The method minimizes human labeling to less than $3\%$ of instances and relies on attention-correctness judgments propagated through a neighbor-aware expansion, enabling robust learning without heavy annotation or compute. Empirical results across Waterbirds, CelebA, ISIC, NICO, and ImageNet-9 show SLIM achieving competitive or superior worst-group accuracy with lower annotation and training costs, along with improved attention alignment as measured by AIoU. The work emphasizes data quality and efficient supervision as a practical path to reliable AI, while noting limitations related to attention-based spuriousness detection and proposing future work to address other spurious feature modalities.

Abstract

Recent studies highlight that deep learning models often learn spurious features mistakenly linked to labels, compromising their reliability in real-world scenarios where such correlations do not hold. Despite the increasing research effort, existing solutions often face two main challenges: they either demand substantial annotations of spurious attributes, or they yield less competitive outcomes with expensive training when additional annotations are absent. In this paper, we introduce SLIM, a cost-effective and performance-targeted approach to reducing spurious correlations in deep learning. Our method leverages a human-in-the-loop protocol featuring a novel attention labeling mechanism with a constructed attention representation space. SLIM significantly reduces the need for exhaustive additional labeling, requiring human input for fewer than 3% of instances. By prioritizing data quality over complicated training strategies, SLIM curates a smaller yet more feature-balanced data subset, fostering the development of spuriousness-robust models. Experimental validations across key benchmarks demonstrate that SLIM competes with or exceeds the performance of leading methods while significantly reducing costs. The SLIM framework thus presents a promising path for developing reliable models more efficiently. Our code is available in https://github.com/xiweix/SLIM.git/.

Figures (11)

• Figure 1: (Left) GradCAM visualizations highlighting model attention on ISIC. (a, b) showcase a model biased towards patches, leading to a correct prediction with wrong reasons in (a) but an incorrect prediction despite correct focus in (b). (c, d) depict a spuriousness-robust model consistently focusing on core features. (Right) The table details the training data distribution by class (benign or malignant) and the presence of color patches, illustrating the imbalance and potential for spurious correlations.
• Figure 2: Overview of SLIM framework, with a data construction pipeline consisting of three phases: (1) Attention Space Construction, creating a space with data features and model attention aligned locally; (2) Attention Annotation and Expansion, where instances sampled from the attention space are labeled by human for attention correctness, and labels are propagated to neighboring instances; and (3) Balanced Data Curation, which filters out instances with incorrect attention, and utilize attention-weighted ($F_A$) and inverse-attention-weighted ($F_{\bar{A}}$) feature vectors to create core and environment feature sets, forming subgroups to assemble a feature-balanced subset for training a spuriousness-robust model.
• Figure 3: Construction of the attention space. In the latent space, feature vector $F$ and model's attribution vector $A$ are extracted, representing input features and model's attention, representatively. By weighting $F$ with $A$, the attention-weighted feature vector $F_A$ emphasizes features of model's top interest. All $F_A$ are then projected to form the attention space featuring locally consistent features and model attention.
• Figure A1: Data distribution in the combination of training and validation set of NICO, with respect to object and context categories.
• Figure A2: Crowdsourcing interface for (a) spuriousness labeling and (b) attention correctness annotation on Waterbirds dataset.

Theorems & Definitions (4)

• definition thmcounterdefinition: Data model.deng2023robust
• lemma thmcounterlemma
• proof
• theorem thmcountertheorem