Improving Robustness to Multiple Spurious Correlations by Multi-Objective Optimization

TL;DR

We address robustness to multiple spurious correlations by partitioning data into $N=2^D$ groups corresponding to $D$ bias types and solving a multi-objective optimization that minimizes the maximum group loss, i.e., a minimax Pareto solution. The method jointly learns model parameters and group-weights through a softmax-weighted sum of group losses and a Pareto-stationarity term that is updated via a Lagrange multiplier every $U$ iterations. A new real-image benchmark, MultiCelebA, is introduced to evaluate debiasing under realistic multi-bias conditions, and experiments across multiple datasets show state-of-the-art performance on several multi-bias benchmarks and strong results on single-bias tasks. The work offers a principled, scalable approach to debiasing with bias-label supervision and demonstrates practical gains in fairness and robustness across diverse visual tasks.

Abstract

We study the problem of training an unbiased and accurate model given a dataset with multiple biases. This problem is challenging since the multiple biases cause multiple undesirable shortcuts during training, and even worse, mitigating one may exacerbate the other. We propose a novel training method to tackle this challenge. Our method first groups training data so that different groups induce different shortcuts, and then optimizes a linear combination of group-wise losses while adjusting their weights dynamically to alleviate conflicts between the groups in performance; this approach, rooted in the multi-objective optimization theory, encourages to achieve the minimax Pareto solution. We also present a new benchmark with multiple biases, dubbed MultiCelebA, for evaluating debiased training methods under realistic and challenging scenarios. Our method achieved the best on three datasets with multiple biases, and also showed superior performance on conventional single-bias datasets.

Figures (10)

• Figure 1: A example of grouping training data with two bias types. Each axis represents each bias type, for which bias-guiding samples make up the majority and bias-conflicting ones hold the minority. In this example, the name of each group indicates if samples of the group has a guiding attribute (G) or a conflicting attribute (C) for gender and age, in respective order.
• Figure 2: The concept of between-group conflicts and model biases. During model parameter updates ($\theta$), the model risks exploiting spurious correlations as shortcuts for classification (red lines). Updating model parameters toward shortcut results in a reduced group-wise loss in the guiding groups but amplifies the loss in the conflicting groups (e.g., CG and CC for shortcut 1), leading to group conflicts. Updating parameters towards cues directly related to the target classification, free from spurious correlations (blue line), offers the only solution to minimize losses across all groups.
• Figure 3: Training set configuration of MultiCelebA in the two-bias setting.
• Figure 4: Change of the group-scaling parameter over time on MultiCelebA in two-bias settings. In the case of GroupDRO, (H) and (L) denote High-cheekbones and Low-cheekbones, respectively.
• Figure 5: Group-wise test accuracy of three different grouping strategies. Lines indicate Unbiased performance, and shaded regions show the lowest and the highest accuracy among the group-wise scores. To ensure fair comparison, the test data are grouped by $\textbf{b}$ and $t$.

Theorems & Definitions (2)

• Definition 3.1: Pareto optimality
• Definition 3.2: Pareto stationarity