Towards Real-world Debiasing: Rethinking Evaluation, Challenge, and Solution

TL;DR

This work investigates spurious correlations under real-world distribution shifts and questions the realism of existing debiasing benchmarks. It introduces a fine-grained bias analysis framework that decomposes bias into feature-level magnitude $\rho^*_a$ via KL divergence and dataset-level prevalence $Prv$, revealing that real-world biases are often low in both dimensions. Building on these insights, the authors present a systematic evaluation framework (RDBench) and two real-world-inspired biases, along with a sparse-bias capturing challenge that existing methods struggle with when no bias labels are available. To address this, they propose Debias in Destruction (DiD), a simple yet effective bias-capture enhancement that destroys target features during bias capture, and validate its effectiveness across eight diverse datasets, improving performance of several baselines and even aiding bias-detection tasks. The results suggest that accounting for sparse, real-world biases is essential for robust debiasing and that DiD provides a practical, broadly applicable improvement for real-world deployments.

Abstract

Spurious correlations in training data significantly hinder the generalization capability of machine learning models when faced with distribution shifts, leading to the proposition of numberous debiasing methods. However, it remains to be asked: \textit{Do existing benchmarks for debiasing really represent biases in the real world?} Recent works attempt to address such concerns by sampling from real-world data (instead of synthesizing) according to some predefined biased distributions to ensure the realism of individual samples. However, the realism of the biased distribution is more critical yet challenging and underexplored due to the complexity of real-world bias distributions. To tackle the problem, we propose a fine-grained framework for analyzing biased distributions, based on which we empirically and theoretically identify key characteristics of biased distributions in the real world that are poorly represented by existing benchmarks. Towards applicable debiasing in the real world, we further introduce two novel real-world-inspired biases to bridge this gap and build a systematic evaluation framework for real-world debiasing, RDBench\footnote{RDBench: Code to be released. Preliminary version in supplementary material for anonimized review.}. Furthermore, focusing on the practical setting of debiasing w/o bias label, we find real-world biases pose a novel \textit{Sparse bias capturing} challenge to the existing paradigm. We propose a simple yet effective approach named Debias in Destruction (DiD), to address the challenge, whose effectiveness is validated with extensive experiments on 8 datasets of various biased distributions.

Figures (9)

• Figure 1: Visualization of the joint distribution for datasets, where the y-axis is the target attribute and the x-axis is the spurious attribute. Figure \ref{['fig:main']}(a) and \ref{['fig:main']}(c) visualise the distribution of existing benchmarks. Figure \ref{['fig:main']}(b) and \ref{['fig:main']}(d) visualize the distribution of real-world datasets. The biased distributions of existing benchmarks and real-world datasets are not alike.
• Figure 2: With our analysis framework, we can see that the bias magnitude and prevalence of real-world datasets are significantly smaller than that of existing benchmarks.
• Figure 3: The bias capture process of biased models on LP and HP datasets. Assuming the red background is spuriously correlated with digit 6, and only the major learning of the biased models is illustrated with arrows. DiD eliminates the undesired learning of BN samples on the LP dataset in Figure \ref{['fig:biasCap']}(a) by destroying the target feature, as shown in Figure \ref{['fig:biasCap']}(b).
• Figure 4: The performance of debiasing methods under various bias magnitudes and prevalence.
• Figure 5: Visualization of the joint distribution for datasets, where the y-axis is the target attribute and the x-axis is the spurious attribute. Figure \ref{['fig:addVis']}(a) visualize the distribution of existing benchmarks. Figure \ref{['fig:addVis']}(b), \ref{['fig:addVis']}(c), \ref{['fig:addVis']}(d), \ref{['fig:addVis']}(e), and \ref{['fig:addVis']}(f) visualize the distribution of real-world datasets. The biased distribution of existing benchmarks and real-world datasets is not alike.