Crucial-Diff: A Unified Diffusion Model for Crucial Image and Annotation Synthesis in Data-scarce Scenarios

TL;DR

Crucial-Diff tackles data scarcity by proposing a domain-agnostic diffusion framework that synthesizes crucial samples and their pixel-level annotations. It introduces two complementary modules: SAFE, which maps reference target features into textual embeddings to guide diffusion, and WASM, which leverages downstream-feedback to generate hard-to-detect (crucial) samples, fused with SAFE outputs via a cross-attention mechanism. The method yields substantial downstream gains across industrial and medical datasets while reducing per-target training requirements, and demonstrates cross-category transferability and real-world applicability. This work advances practical data augmentation for detection and segmentation in scarce-data regimes, with strong quantitative and qualitative evidence and publicly available code.

Abstract

The scarcity of data in various scenarios, such as medical, industry and autonomous driving, leads to model overfitting and dataset imbalance, thus hindering effective detection and segmentation performance. Existing studies employ the generative models to synthesize more training samples to mitigate data scarcity. However, these synthetic samples are repetitive or simplistic and fail to provide "crucial information" that targets the downstream model's weaknesses. Additionally, these methods typically require separate training for different objects, leading to computational inefficiencies. To address these issues, we propose Crucial-Diff, a domain-agnostic framework designed to synthesize crucial samples. Our method integrates two key modules. The Scene Agnostic Feature Extractor (SAFE) utilizes a unified feature extractor to capture target information. The Weakness Aware Sample Miner (WASM) generates hard-to-detect samples using feedback from the detection results of downstream model, which is then fused with the output of SAFE module. Together, our Crucial-Diff framework generates diverse, high-quality training data, achieving a pixel-level AP of 83.63% and an F1-MAX of 78.12% on MVTec. On polyp dataset, Crucial-Diff reaches an mIoU of 81.64% and an mDice of 87.69%. Code is publicly available at https://github.com/JJessicaYao/Crucial-diff.

Figures (9)

• Figure 1: Comparison of previous synthesis methods and our Crucial-Diff. (a) In data distribution comparison, our method generates images (purple points) that are more challenging for downstream tasks, while previous methods produce samples (yellow points) that closely resemble real images (pink points) and easy detection. (b) The visualization comparison shows that our samples are less identifiable, while prior methods replicate target patterns from the original dataset, with white dashed boxes indicating target areas. (c) The downstream task metrics comparison highlights the pixel-level performance of a U-Net model trained on synthetic images. As data volume increases, our method consistently improves results, while previous methods decline, emphasizing the importance of generating crucial data.
• Figure 2: Overview of Crucial-Diff framework. (a) Inference pipeline of Crucial-Diff, which utilizes a mask and a background image as inputs, with a target image to guide the synthesis process. Crucial-Diff generates critical samples and extracts annotations from attention layers. (b) Training pipeline of Crucial-Diff, which consists of two stages. Stage 1 trains a Scene Agnostic Feature Extractor (SAFE) module to convert target images into textual features. Stage 2 trains a Weakness Aware Sample Miner (WASM) module to extract features that challenge downstream models in detecting targets from background, then fuse extracted features with SAFE in cross-attention layer.
• Figure 3: Visualization of iterative refinement of the cross-attention layer. The first row displays cross-attention maps when the iteration is 0, 1, 3, 5 and 7, while the second row shows the corresponding pixel-level annotations $\hat{m}$.
• Figure 4: Visualization comparison of synthetic images generated by Crucial-Diff and other methods. (a) is MVTec result, (b) is Real-IAD dataset result and (c) is polyp dataset result with the annotations of synthetic targets at bottom right of each image. Our generated samples exhibit features like seamless background fusion, surface irregularities or subtle color variations that effectively disrupt downstream models and significantly increase the difficulty of target distinction.
• Figure 5: T-SNE visualization of different targets. The pink triangle represents a real image with target, the green circle represents a real image without target, the yellow pentagram represents an easy synthetic image with target, and the purple diamond represents a crucial synthetic image with target.