Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model

TL;DR

The empirical results demonstrate that Diff-Mix achieves a better balance between faith-fulness and diversity, leading to a marked improvement in performance across diverse image classification scenarios, including few-shot, conventional, and long-tail classifications for domain-specific datasets.

Abstract

Text-to-image (T2I) generative models have recently emerged as a powerful tool, enabling the creation of photo-realistic images and giving rise to a multitude of applications. However, the effective integration of T2I models into fundamental image classification tasks remains an open question. A prevalent strategy to bolster image classification performance is through augmenting the training set with synthetic images generated by T2I models. In this study, we scrutinize the shortcomings of both current generative and conventional data augmentation techniques. Our analysis reveals that these methods struggle to produce images that are both faithful (in terms of foreground objects) and diverse (in terms of background contexts) for domain-specific concepts. To tackle this challenge, we introduce an innovative inter-class data augmentation method known as Diff-Mix (https://github.com/Zhicaiwww/Diff-Mix), which enriches the dataset by performing image translations between classes. Our empirical results demonstrate that Diff-Mix achieves a better balance between faithfulness and diversity, leading to a marked improvement in performance across diverse image classification scenarios, including few-shot, conventional, and long-tail classifications for domain-specific datasets.

Figures (20)

• Figure 1: Strategies to expand domain-specific datasets for improved classification are varied. Row 1 illustrates vanilla distillation from a pretrained text-to-image (T2I) model, which carries the risk of generating outputs with reduced faithfulness. Intra-class augmentation, depicted in Row 2, tends to yield samples with limited diversity to maintain high fidelity to the original class. Our proposed method, showcased in Rows 3 and 4, adopts an inter-class augmentation strategy. This involves introducing edits to a reference image using guidance of another class, which significantly enriches the dataset with a greater diversity of samples.
• Figure 2: Examples of "American Three toed Woodpecker". (a) Real images from the training set. (b-d) synthetic images generated using different fine-tuned models with the same number of fine-tuning steps. TI+DB indicates both text embedding and U-Net are fine-tuned. TI+DB achieves a more faithful output compared to DB alone (check the head and wing patterns of the birds).
• Figure 2: Long-tail classification in CUB-LT.
• Figure 3: Fine-tuning framework of Diff-Mix operates as follows: Initially, we replace the class name with a structured identifier formatted as "[V$^i$] [metaclass]", thereby sidestepping the need for specific terminological expressions. Next, we engage in joint fine-tuning of these identifiers and the low-rank residues (LoRA) of U-Net to capture the domain-specific distribution.
• Figure 4: Examples of images translated using Diff-Mix and Diff-Aug across various strengths. Diff-Aug employs the same target and reference image classes, typically resulting in subtle modifications. Diff-Mix progressively adjusts the foreground to align with the target class as the translation strength increases, while preserving the background layout from the reference image.