Empowering Vision Transformers with Multi-Scale Causal Intervention for Long-Tailed Image Classification

TL;DR

This paper tackles long-tail image classification in Vision Transformers by identifying the failure of existing causal methods to leverage fine-grained features. It introduces TSCNet, a two-stage framework consisting of Hierarchical Causal Representation Learning (HCRL) to remove semantic bias at patch and feature levels, and Counterfactual Logits Bias Calibration (CLBC) to adjust decision boundaries via counterfactual augmentation. The method employs a confounder dictionary, NWGM approximation, and Fourier-based counterfactuals with adaptive strength, framed by backdoor adjustments. Empirical results on CIFAR100-LT and VireoFood-172 show that TSCNet consistently improves tail-class performance across backbones, especially ViT-based models, while preserving head-class accuracy, underscoring its practical potential for robust long-tail vision systems.

Abstract

Causal inference has emerged as a promising approach to mitigate long-tail classification by handling the biases introduced by class imbalance. However, along with the change of advanced backbone models from Convolutional Neural Networks (CNNs) to Visual Transformers (ViT), existing causal models may not achieve an expected performance gain. This paper investigates the influence of existing causal models on CNNs and ViT variants, highlighting that ViT's global feature representation makes it hard for causal methods to model associations between fine-grained features and predictions, which leads to difficulties in classifying tail classes with similar visual appearance. To address these issues, this paper proposes TSCNet, a two-stage causal modeling method to discover fine-grained causal associations through multi-scale causal interventions. Specifically, in the hierarchical causal representation learning stage (HCRL), it decouples the background and objects, applying backdoor interventions at both the patch and feature level to prevent model from using class-irrelevant areas to infer labels which enhances fine-grained causal representation. In the counterfactual logits bias calibration stage (CLBC), it refines the optimization of model's decision boundary by adaptive constructing counterfactual balanced data distribution to remove the spurious associations in the logits caused by data distribution. Extensive experiments conducted on various long-tail benchmarks demonstrate that the proposed TSCNet can eliminate multiple biases introduced by data imbalance, which outperforms existing methods.

Figures (8)

• Figure 1: The illustration of the proposed TSCNet. It removes semantic bias through hierarchical causal intervention to enhance the causal representation of tail classes. In the second stage, it adaptively calibrates logit bias through counterfactual intervention.
• Figure 2: The example of Similar-FP and Non-similar-FP on (a), the error confusion analysis of the existing causal method xERM on ViT on (b), the bias correction mechanism of causal methods TDE on (c), the difference in counterfactual estimation logits class consistency long-tail bias between the method TDE on ViT and CNNs. on (d).
• Figure 3: The Causal view of Long-tailed Image Classification.
• Figure 4: Illustration of the proposed TSCNet. It contains two main stages: HCRL and CLBC. The former introduces class-independent semantic information and performs backdoor adjustments to enhance the model’s fine-grained causal representation for tail classes $S\not\rightarrow X$. The latter generates counterfactual distribution to calibrate logits bias and model category relationships $D\not\rightarrow X$.
• Figure 5: Comparison of our adaptive adjustment of the enhancement parameter $L_c^e$ with using fixed parameters 0.36 and 0.64 in terms of performance, positive values indicate that our parameter adjustment performs better in this category.