Semantics-Aware Generative Latent Data Augmentation for Learning in Low-Resource Domains

TL;DR

Semantics-Aware Generative Latent Data Augmentation (GeLDA) addresses data scarcity and label imbalance by performing data augmentation in a foundation-model–derived latent space using conditional diffusion. By conditioning on semantic embeddings and subdomain information, GeLDA transfers cues from high-resource to low-resource classes and subdomains, while selecting an appropriate latent layer to balance diversity and learning capacity. The approach yields state-of-the-art tail-class performance in long-tailed image classification (ImageNet-LT) and a notable improvement in zero-shot language-specific speech emotion recognition, demonstrating cross-modal effectiveness and efficiency. GeLDA reduces data and computation requirements relative to input-space DA, enabling practical deployment in real-world low-resource scenarios and offering a generalizable framework for SEM and subdomain-aware augmentation.

Abstract

Despite strong performance in data-rich regimes, deep learning often underperforms in the data-scarce settings common in practice. While foundation models (FMs) trained on massive datasets demonstrate strong generalization by extracting general-purpose features, they can still suffer from scarce labeled data during downstream fine-tuning. To address this, we propose GeLDA, a semantics-aware generative latent data augmentation framework that leverages conditional diffusion models to synthesize samples in an FM-induced latent space. Because this space is low-dimensional and concentrates task-relevant information compared to the input space, GeLDA enables efficient, high-quality data generation. GeLDA conditions generation on auxiliary feature vectors that capture semantic relationships among classes or subdomains, facilitating data augmentation in low-resource domains. We validate GeLDA in two large-scale recognition tasks: (a) in zero-shot language-specific speech emotion recognition, GeLDA improves the Whisper-large baseline's unweighted average recall by 6.13%; and (b) in long-tailed image classification, it achieves 74.7% tail-class accuracy on ImageNet-LT, setting a new state-of-the-art result.

Figures (7)

• Figure 1: Example of the GeLDA framework with an $L=3$ task adapter in $\mathcal{Z}^{(1)}$. (a) Repurposing the FM into a downstream recognition model via adapter layers. (b) Training a latent diffusion model to synthesize features in $\mathcal{Z}^{(1)}$, conditioned on augmented label information $u(\gamma)$ to transfer cues to the low-resource class $\gamma$ from related high-resource classes. (c) Fine-tuning the model using synthesized and ground-truth samples in $\mathcal{Z}^{(1)}$.
• Figure 2: Illustration of the proposed GeLDA in the feature space. (a) Original data distribution $\mathcal{X}$ with a complex decision boundary. (b) More separable class distributions in the learned feature space $\mathcal{Z}$. (c) Few samples from the low-resource subset $\mathcal{D}^{(\kappa)}$. Contours are from the unknown ground-truth distribution of subdomain $\kappa$. (d) GeLDA's feature-space data augmentation results $\bar{z}$, whose sample distribution resemble the distribution of $k'$. $\kappa$ and $k'$ are semantically similar subdomains.
• Figure 3: Graph of GeLDA performance across various diffusion model sizes. Results use Whisper-large as the backbone FM with 200 augmented samples per emotion.
• Figure 4: GeLDA performance with varying numbers of augmented samples using Whisper-large as the backbone FM.
• Figure 5: Examples of t-SNE plots of generated embeddings from GeLDA and GT embeddings from the test set for the SER task. From left to right, the plots correspond to the embedding spaces $\mathcal{Z}^{(0)}$, $\mathcal{Z}^{(1)}$, and $\mathcal{Z}^{(2)}$. All results are shown for the Italian case, and the foundation model is Whisper-large.