Fine-Grained Bias Exploration and Mitigation for Group-Robust Classification

TL;DR

The paper tackles robust group generalization under spurious correlations without bias annotations. It introduces Bias Exploration via Overfitting (BEO) to infer latent bias subgroups from overfitting behavior, and Fine-Grained Class-Conditional Distribution Balancing (FG-CCDB) to perform group-wise, closed-form distribution matching using a mixture of bias groups. The method demonstrates that BEO serves as a strong proxy for ground-truth bias annotations and, when combined with FG-CCDB, achieves performance on par with bias-supervised approaches in binary tasks and substantially outperforms them in highly biased multi-class settings. Overall, the approach provides annotation-free, scalable bias mitigation with substantial practical impact for improving group robustness in diverse domains.

Abstract

Achieving group-robust generalization in the presence of spurious correlations remains a significant challenge, particularly when bias annotations are unavailable. Recent studies on Class-Conditional Distribution Balancing (CCDB) reveal that spurious correlations often stem from mismatches between the class-conditional and marginal distributions of bias attributes. They achieve promising results by addressing this issue through simple distribution matching in a bias-agnostic manner. However, CCDB approximates each distribution using a single Gaussian, which is overly simplistic and rarely holds in real-world applications. To address this limitation, we propose a novel method called Bias Exploration via Overfitting (BEO), which captures each distribution in greater detail by modeling it as a mixture of latent groups. Building on these group-level descriptions, we introduce a fine-grained variant of CCDB, termed FG-CCDB, which performs more precise distribution matching and balancing within each group. Through group-level reweighting, FG-CCDB learns sample weights from a global perspective, achieving stronger mitigation of spurious correlations without incurring substantial storage or computational costs. Extensive experiments demonstrate that BEO serves as a strong proxy for ground-truth bias annotations and can be seamlessly integrated with bias-supervised methods. Moreover, when combined with FG-CCDB, our method performs on par with bias-supervised approaches on binary classification tasks and significantly outperforms them in highly biased multi-class scenarios.

Figures (5)

• Figure 1: The framework of our BEO. The initial biased learning stage produces a partition (indicated by the black and red lines) where only a small portion of minority groups is identified. Through bias enhancement learning, the framework progressively uncovers most of the minority groups.
• Figure 2: Left: the framework of our FG-CCDB. No optimization process is needed. Right: the joint distribution ${\bf G}$ of bias and class labels estimated by our method.
• Figure 3: The effect of FG-CCDB sample reweighting in reshaping the data distribution and mitigating spurious correlations across four datasets.
• Figure 4: The group prediction accuracy (left) and the worst-group accuracy (right) along the repeating of the "bias enhancement learning" procedure.
• Figure 5: The distribution of the sample weights assigned by FG-CCDB within each group on four datasets.