From Fake to Real: Pretraining on Balanced Synthetic Images to Prevent Spurious Correlations in Image Recognition

TL;DR

This work tackles spurious correlations in image recognition arising from biased real-world datasets by highlighting a distributional gap between real and synthetic data. It theoretically shows that augmenting biased data with synthetic samples induces a new $(B,G)$ bias, prompting a two-stage solution: pretrain on balanced synthetic data to learn unbiased representations, then fine-tune on real data with standard ERM or bias-mitigation losses. The From Fake to Real (FFR) pipeline avoids exposing the model to real-synthetic distribution differences during joint training, leading to improved worst-group accuracy by up to 20 percentage points across multiple datasets. Empirical results, ablations, and qualitative analyses support FFR's robustness to bias and its compatibility with existing bias-mitigation methods, though it relies on balanced synthetic pretraining and acknowledges potential generative-model biases.

Abstract

Visual recognition models are prone to learning spurious correlations induced by a biased training set where certain conditions $B$ (\eg, Indoors) are over-represented in certain classes $Y$ (\eg, Big Dogs). Synthetic data from off-the-shelf large-scale generative models offers a promising direction to mitigate this issue by augmenting underrepresented subgroups in the real dataset. However, by using a mixed distribution of real and synthetic data, we introduce another source of bias due to distributional differences between synthetic and real data (\eg synthetic artifacts). As we will show, prior work's approach for using synthetic data to resolve the model's bias toward $B$ do not correct the model's bias toward the pair $(B, G)$, where $G$ denotes whether the sample is real or synthetic. Thus, the model could simply learn signals based on the pair $(B, G)$ (\eg, Synthetic Indoors) to make predictions about $Y$ (\eg, Big Dogs). To address this issue, we propose a simple, easy-to-implement, two-step training pipeline that we call From Fake to Real (FFR). The first step of FFR pre-trains a model on balanced synthetic data to learn robust representations across subgroups. In the second step, FFR fine-tunes the model on real data using ERM or common loss-based bias mitigation methods. By training on real and synthetic data separately, FFR does not expose the model to the statistical differences between real and synthetic data and thus avoids the issue of bias toward the pair $(B, G)$. Our experiments show that FFR improves worst group accuracy over the state-of-the-art by up to 20\% over three datasets. Code available: \url{https://github.com/mqraitem/From-Fake-to-Real}

Figures (7)

• Figure 1: Comparison of RISE Petsiuk2018rise saliency maps produced for a model trained to predict the attribute Smiling $Y$ given a bias toward Gender $B$ (Most Women are not Smiling) using: (a) prior work in Synthetic Augmentation Ramaswamy_2021_CVPRMondal2023MinorityOF, which do not address the unexpected bias toward the pair (Gender $B$, Data Source $G$), e.g., (Female-Synthetic vs Female-Real) and, thus, use spurious features leading to an incorrect prediction (not smiling) (b) our approach FFR where the Synthetic and Real Data are separated into two training stages, thereby mitigating the bias suffered by prior work enabling us to learn the correct features (Mouth) and correctly predict Smiling.
• Figure 2: An overview of From Fake to Real (FFR) that incorporates synthetic data to mitigate bias. In Stage 1, we pretrain on a balanced synthetic dataset where we learn robust representations across subgroups. In Stage 2, we fine-tune the model on real data using ERM or common synthetic-data-free bias mitigation methods. By training on real and synthetic data separately, FFR does not expose the model to the statistical differences between real and synthetic data and thus avoids the issue of bias between the two data sources. Refer to Section \ref{['sec:ffr']} for further discussion.
• Figure 3: Comparison of performance between the effect: (None) no synthetic data is used, (USB) synthetic data is used to uniformly balance the distribution (extension of prior work on imbalanced classification Mondal2023MinorityOF), (ASB) balanced synthetic data is added to the real dataset Ramaswamy_2021_CVPR and (FFR) our method where pretrain on balanced synthetic data and fine tune on real data. Models are trained with ERM. Note how our method either matches or improves the performance of prior work augmentation methods. Refer to Section \ref{['sec:systemic_analysis_exp']} for discussion.
• Figure 4: Comparing the performance of synthetic-data-free bias mitigation methods, namely GroupDRO sagawa2019distributionally, Resampling, and Deep Feature Reweighting (DFR) kirichenko2022last with no synthetic augmentation (None) as well as with synthetic augmentation using prior work methods (USB Mondal2023MinorityOF and ASB Ramaswamy_2021_CVPR) and our method FFR. Performance is averaged across three datasets and five bias ratios. Note how our method (FFR) in column four is best at improving the performance of non-synthetic-data augmentation methods. Refer to \ref{['sec:harder_analysis_exp']} for further discussion.
• Figure 5: Comparing the projections of Real vs. synthetic Data using t-SNE van2008visualizing with prior work synthetic augmentation (USB Mondal2023MinorityOF and ASB Ramaswamy_2021_CVPR) and our synthetic Augmentation method (FFR). Note how our method (FFR) is the best method for projecting Real and Synthetic data close to each other. This is likely because FFR is less impacted by the bias between real and synthetic data and, thus, is posed to learn best from the two data sources. Refer to Section \ref{['sec:real-synth-bias-mitigation']} for further discussion.

Theorems & Definitions (5)

• theorem \@thmcountertheorem
• lemma \@thmcounterlemma
• proof
• theorem \@thmcountertheorem
• proof