SGIA: Enhancing Fine-Grained Visual Classification with Sequence Generative Image Augmentation

TL;DR

The paper tackles fine-grained visual classification (FGVC) where data scarcity and subtle inter-class differences hinder performance. It introduces Sequence Generative Image Augmentation (SGIA) based on a Sequence Latent Diffusion Model (SLDM) and Bridging Transfer Learning (BTL) to generate diverse, realistic image sequences that preserve discriminative features. A balancing strategy with parameter $\alpha$ integrates real and synthetic data, while BTL enables two-stage transfer learning to bridge domain gaps between real data and augmented samples. Across three FGVC datasets and multiple backbones, SGIA consistently improves accuracy over baselines and conventional GIA, achieving new state-of-the-art results on CUB-200-2011 (including a 0.5% gain with optimized pretraining). The work demonstrates that sequence-based diffusion with careful data balancing and transfer learning can dramatically enhance FGVC performance, especially in few-shot regimes, and offers practical guidance for applying generative augmentations in real-world FGVC pipelines.

Abstract

In Fine-Grained Visual Classification (FGVC), distinguishing highly similar subcategories remains a formidable challenge, often necessitating datasets with extensive variability. The acquisition and annotation of such FGVC datasets are notably difficult and costly, demanding specialized knowledge to identify subtle distinctions among closely related categories. Our study introduces a novel approach employing the Sequence Latent Diffusion Model (SLDM) for augmenting FGVC datasets, called Sequence Generative Image Augmentation (SGIA). Our method features a unique Bridging Transfer Learning (BTL) process, designed to minimize the domain gap between real and synthetically augmented data. This approach notably surpasses existing methods in generating more realistic image samples, providing a diverse range of pose transformations that extend beyond the traditional rigid transformations and style changes in generative augmentation. We demonstrate the effectiveness of our augmented dataset with substantial improvements in FGVC tasks on various datasets, models, and training strategies, especially in few-shot learning scenarios. Our method outperforms conventional image augmentation techniques in benchmark tests on three FGVC datasets, showcasing superior realism, variability, and representational quality. Our work sets a new benchmark and outperforms the previous state-of-the-art models in classification accuracy by 0.5% for the CUB-200-2011 dataset and advances the application of generative models in FGVC data augmentation.

Figures (5)

• Figure 1: Illustration of synthetic image quality. The left image is from the CUB-2011-200 dataset. The four images on the right are synthetic ones generated from the original.
• Figure 2: Two-phase neural network training framework. The process begins with encoding images with video motion and semantic features to guide the SLDM in the denoising phase. A Balancing Sampler then integrates augmented data with original data for the transfer learning of the bridging model. Finally, this model is fine-tuned on the original dataset to complete the classification model.
• Figure 3: FGVC accuracies on CUB-200-2011 dataset wah2011caltech of the proposed SGIA vs. different configurations of augmentation probability $\alpha$ and augmentations per sample $M$, and comparison with GIA (real guidance he2023is).
• Figure 4: Generatied samples from GIA and SGIA.
• Figure 5: Negative samples from SGIA. The "Original" column displays real images from the three FGVC datasets. The "Augmentation" column shows negative samples generated by SGIA, characterized by less distinguishable features or lower image quality.