Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Salient Concept-Aware Generative Data Augmentation

Tianchen Zhao, Xuanbai Chen, Zhihua Li, Jun Fang, Dongsheng An, Xiang Xu, Zhuowen Tu, Yifan Xing

TL;DR

This work tackles the fidelity-diversity trade-off in generative data augmentation for fine-grained visual classification by introducing a domain-specific Salient Concept-Aware (SCA) image embedding paired with a diffusion-based synthesis model. The SCA embedding is trained with an angular-margin loss to emphasize the salient concept and suppress irrelevant background information, while the synthesis model fuses image and text prompts through cross-attention and LoRA adapters. Training occurs in two stages: first the SCA embedding is learned, then the synthesis model is fine-tuned with frozen embeddings, enabling faithful yet diverse synthetic data generation guided by contextually rich prompts. Across eight FGVC datasets, this approach yields state-of-the-art gains, especially in long-tail and out-of-distribution settings, demonstrating improved classifier robustness with a practical training cost and scalable data generation.

Abstract

Recent generative data augmentation methods conditioned on both image and text prompts struggle to balance between fidelity and diversity, as it is challenging to preserve essential image details while aligning with varied text prompts. This challenge arises because representations in the synthesis process often become entangled with non-essential input image attributes such as environmental contexts, creating conflicts with text prompts intended to modify these elements. To address this, we propose a personalized image generation framework that uses a salient concept-aware image embedding model to reduce the influence of irrelevant visual details during the synthesis process, thereby maintaining intuitive alignment between image and text inputs. By generating images that better preserve class-discriminative features with additional controlled variations, our framework effectively enhances the diversity of training datasets and thereby improves the robustness of downstream models. Our approach demonstrates superior performance across eight fine-grained vision datasets, outperforming state-of-the-art augmentation methods with averaged classification accuracy improvements by 0.73% and 6.5% under conventional and long-tail settings, respectively.

Salient Concept-Aware Generative Data Augmentation

TL;DR

This work tackles the fidelity-diversity trade-off in generative data augmentation for fine-grained visual classification by introducing a domain-specific Salient Concept-Aware (SCA) image embedding paired with a diffusion-based synthesis model. The SCA embedding is trained with an angular-margin loss to emphasize the salient concept and suppress irrelevant background information, while the synthesis model fuses image and text prompts through cross-attention and LoRA adapters. Training occurs in two stages: first the SCA embedding is learned, then the synthesis model is fine-tuned with frozen embeddings, enabling faithful yet diverse synthetic data generation guided by contextually rich prompts. Across eight FGVC datasets, this approach yields state-of-the-art gains, especially in long-tail and out-of-distribution settings, demonstrating improved classifier robustness with a practical training cost and scalable data generation.

Abstract

Recent generative data augmentation methods conditioned on both image and text prompts struggle to balance between fidelity and diversity, as it is challenging to preserve essential image details while aligning with varied text prompts. This challenge arises because representations in the synthesis process often become entangled with non-essential input image attributes such as environmental contexts, creating conflicts with text prompts intended to modify these elements. To address this, we propose a personalized image generation framework that uses a salient concept-aware image embedding model to reduce the influence of irrelevant visual details during the synthesis process, thereby maintaining intuitive alignment between image and text inputs. By generating images that better preserve class-discriminative features with additional controlled variations, our framework effectively enhances the diversity of training datasets and thereby improves the robustness of downstream models. Our approach demonstrates superior performance across eight fine-grained vision datasets, outperforming state-of-the-art augmentation methods with averaged classification accuracy improvements by 0.73% and 6.5% under conventional and long-tail settings, respectively.
Paper Structure (21 sections, 4 equations, 4 figures, 13 tables)

This paper contains 21 sections, 4 equations, 4 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our Approach
  4. Salient Concept-Aware Embedding Model
  5. Synthesis Model
  6. Synthesis for Analysis
  7. Experiments
  8. Experiments
  9. Experimental Setting
  10. Performance of Generative Data Augmentation
  11. Ablation Studies
  12. Conclusion
  13. Model Preliminaries
  14. More Training Details
  15. Long-Tail Recognition Methods
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Comparison between our proposed data augmentation method with DA-Fusion trabucco2023effective. DA-Fusion uses Textual Inversion gal2022image to learn the word embedding of the ladybug from the image prompt. But in this example, the generated outputs skew heavily toward a green color palette, failing to both preserve the ladybug's characteristics (black body with red spots) and align with the text prompts. Our method is designed with in-context understanding of the task: it selectively encodes salient concept (the ladybug) and effectively reduce the influence of irrelevant information (the greens and leaves) from the prompt, generating images with focused diversity and thereby effectively improves the robustness of the downstream models.
  • Figure 2: Comparison of generated images conditioned on various image prompts and null text prompt. Our model focuses on the fly in the image, evidenced by the high cosine similarity 0.82 between the segmented fly image (d) and the original fly image (e), allowing our synthesis model to generate high-fidelity fly images (ab). Foundation model like DINOv2 encodes lots of information about the background, evidenced by the high cosine similarity 0.83 between the embeddings of the original fly image (e) and the image with the fly removed (f), from which the model can reconstruct the background (i). Although DINOv2 works well on segmented images with the salient object to generate image (g), it introduces additional complexity in the pipeline, which is not suited for data augmentation purposes, and does not work well with abstract concepts without well-defined boundaries.
  • Figure 3: Our proposed framework. The SCA embedding model captures the salient concept and projects it with text embedding into cross-attention layers of U-Net. LoRA is applied to self-attention layers.
  • Figure 4: Impact of synthetic-to-real data ratio $K$ and synthetic sampling probability $P$ on Top-1 classification accuracy. Performance improves when a balanced proportion of synthetic data is used, but degrades when synthetic samples dominate. Increasing $P$ improves dataset diversity, but the benefits diminish as $P$ grows.