Does Feasibility Matter? Understanding the Impact of Feasibility on Synthetic Training Data

TL;DR

This work introduces VariReal, a minimal-change editing pipeline that generates feasible and infeasible synthetic edits of real images across background, color, and texture to study their impact on CLIP-based classifiers trained with LoRA. By leveraging GPT-4 for prompt generation, diffusion- and ControlNet–based editing, and Llava-Next for automatic filtering, the authors create controlled synthetic datasets and analyze the role of feasibility. Across three fine-grained datasets, background edits consistently improve performance, while foreground edits offer limited or mixed benefits, and mixing feasible/infeasible data can provide complementary gains. The findings suggest that feasibility alone is not a strong determinant of downstream performance for object-centric tasks, guiding synthetic-data design toward targeted background variation and cautious use of foreground edits. Overall, the study provides nuanced insights into how synthetic data can augment real data for improved robustness without requiring strict feasibility constraints.

Abstract

With the development of photorealistic diffusion models, models trained in part or fully on synthetic data achieve progressively better results. However, diffusion models still routinely generate images that would not exist in reality, such as a dog floating above the ground or with unrealistic texture artifacts. We define the concept of feasibility as whether attributes in a synthetic image could realistically exist in the real-world domain; synthetic images containing attributes that violate this criterion are considered infeasible. Intuitively, infeasible images are typically considered out-of-distribution; thus, training on such images is expected to hinder a model's ability to generalize to real-world data, and they should therefore be excluded from the training set whenever possible. However, does feasibility really matter? In this paper, we investigate whether enforcing feasibility is necessary when generating synthetic training data for CLIP-based classifiers, focusing on three target attributes: background, color, and texture. We introduce VariReal, a pipeline that minimally edits a given source image to include feasible or infeasible attributes given by the textual prompt generated by a large language model. Our experiments show that feasibility minimally affects LoRA-fine-tuned CLIP performance, with mostly less than 0.3% difference in top-1 accuracy across three fine-grained datasets. Also, the attribute matters on whether the feasible/infeasible images adversarially influence the classification performance. Finally, mixing feasible and infeasible images in training datasets does not significantly impact performance compared to using purely feasible or infeasible datasets.

Figures (20)

• Figure 1: We propose VariReal, a pipeline for minimal-change editing of real images, enabling isolation of target attributes in three categories: background, color, and texture. We compare images generated by VariReal to those produced by prior text-guided editing methods instructpix2pixfpe, examining both feasible and infeasible attributes. The editing prompts are provided below each generated image.
• Figure 2: We compare images generated by various candidate methods: Inpainting model sdinpaint alone, ControlNet controlnet alone, Inpainting model with Real Prior, ControlNet with Raw Prior or Real Prior, and our final results for three attribute modifications. The first two columns illustrate the priors used (Raw Prior and Real Prior), and generation prompts used are listed beneath each image.
• Figure 3: An overview of VariReal pipeline. Minimal-change steps for background, color, and texture are highlighted in green, pink, and grey, respectively. Real images are processed to generate guidance maps (e.g., masks, Canny edges) for Inpainting and ControlNet. GPT-4 generates feasible and infeasible prompts ($P_{f}$ and $P_{if}$), which guide color retrieval or prior image generation via Stable Diffusion. These Real Priors, combined with masks and prompts, are input to the inpainting model. For color and texture, ControlNet with Canny conditioning ensures precise foreground shapes. A final refinement step produces the optimal output for each setting.
• Figure 4: Selected generation results from the three datasets. Only target prompt keywords are shown; detailed background and texture descriptions are omitted. Please zoom in for visual details.
• Figure 5: The FID score settings compared using feasible and infeasible settings across different datasets.