Mitigating Spurious Correlations with Causal Logit Perturbation

TL;DR

The paper tackles the problem of spurious correlations in deep learning by introducing Causal Logit Perturbation (CLP), a framework that learns sample-specific logit perturbations to suppress non-causal background cues. CLP combines a perturbation network (a two-layer MLP) with metadata augmented via counterfactual and factual examples guided by human causal knowledge, and feeds ten training-dynamics features into the perturbation network within an online meta-learning loop. The approach yields state-of-the-art results across four biased-learning scenarios—long-tailed, noisy labels, subpopulation shifts, and generalized long-tail—while visualization and ablation studies corroborate that the perturbations redirect attention to causal features and disrupt spurious associations. Overall, CLP offers a practical, generalizable mechanism to improve robustness and fairness in vision models by explicitly modeling causal interventions at the logit level.

Abstract

Deep learning has seen widespread success in various domains such as science, industry, and society. However, it is acknowledged that certain approaches suffer from non-robustness, relying on spurious correlations for predictions. Addressing these limitations is of paramount importance, necessitating the development of methods that can disentangle spurious correlations. {This study attempts to implement causal models via logit perturbations and introduces a novel Causal Logit Perturbation (CLP) framework to train classifiers with generated causal logit perturbations for individual samples, thereby mitigating the spurious associations between non-causal attributes (i.e., image backgrounds) and classes.} {Our framework employs a} perturbation network to generate sample-wise logit perturbations using a series of training characteristics of samples as inputs. The whole framework is optimized by an online meta-learning-based learning algorithm and leverages human causal knowledge by augmenting metadata in both counterfactual and factual manners. Empirical evaluations on four typical biased learning scenarios, including long-tail learning, noisy label learning, generalized long-tail learning, and subpopulation shift learning, demonstrate that CLP consistently achieves state-of-the-art performance. Moreover, visualization results support the effectiveness of the generated causal perturbations in redirecting model attention towards causal image attributes and dismantling spurious associations.

Figures (9)

• Figure 1: Illustration for spurious correlation resulting from a rare background occurrence. Dogs depicted in water are easy to be misclassified as drakes, primarily due to the infrequency of water backgrounds in dog class.
• Figure 2: The causal graph before (a) and after (b) the intervention process of $\boldsymbol{u}_{z}$. By intervening on $\boldsymbol{u}_{z}$, the backdoor path $z\!\leftarrow\! c\!\rightarrow\!y$ is broken. The observed and unobserved variables are represented by white and grey nodes, respectively, while the intervened variable is depicted using a square. The variables follow the formulas on the right.
• Figure 3: The overall structure of CLP, which consists of four main components: the metadata augmentation module, the backbone classifier, the training characteristics module, and the perturbation network which generates sample-wise logit perturbations. The black and red lines indicate the learning loops of the backbone classifier and the perturbation network, respectively.
• Figure 4: Class-wise accuracy on CIFAR10-LT with imbalance ratios of 10:1 (a) and 100:1 (b). Class-wise accuracy on CIFAR100-LT with imbalance ratios of 10:1 (c) and 100:1 (d). From "C1" to "C10", the number of samples varies from small to large.
• Figure 5: Ratio of samples in CIFAR10 where perturbations from noise and clean samples result in an increase in losses during training. (a) and (b) respectively illustrate the scenarios with 20% and 40% flip noise. The proportion of clean samples with increased losses is significantly higher than that of noisy samples, thereby reducing the adverse impact of noisy samples on model training.