Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CAARMA: Class Augmentation with Adversarial Mixup Regularization

Massa Baali, Xiang Li, Hao Chen, Syed Abdul Hannan, Rita Singh, Bhiksha Raj

TL;DR

CAARMA tackles limited class diversity in zero-shot speaker verification by generating synthetic speaker identities via embedding-space SL-Mixup and refining them with an adversarial mixup discriminator. The approach expands effective training classes without extra real data and preserves inference cost, yielding up to an ~8% improvement over baselines in SV and gains in emotion and distance tasks. Key contributions include the SL-Mixup mechanism, a HuBERT-based mixup discriminator, and a scalable training framework validated on VoxCeleb1/2 and Dynamic-SUPERB. The work demonstrates robust generalization across tasks and provides public code for reproducibility and broader adoption in zero-shot speech analysis.

Abstract

Speaker verification is a typical zero-shot learning task, where inference of unseen classes is performed by comparing embeddings of test instances to known examples. The models performing inference must hence naturally generate embeddings that cluster same-class instances compactly, while maintaining separation across classes. In order to learn to do so, they are typically trained on a large number of classes (speakers), often using specialized losses. However real-world speaker datasets often lack the class diversity needed to effectively learn this in a generalizable manner. We introduce CAARMA, a class augmentation framework that addresses this problem by generating synthetic classes through data mixing in the embedding space, expanding the number of training classes. To ensure the authenticity of the synthetic classes we adopt a novel adversarial refinement mechanism that minimizes categorical distinctions between synthetic and real classes. We evaluate CAARMA on multiple speaker verification tasks, as well as other representative zero-shot comparison-based speech analysis tasks and obtain consistent improvements: our framework demonstrates a significant improvement of 8\% over all baseline models. The code is available at: https://github.com/massabaali7/CAARMA/

CAARMA: Class Augmentation with Adversarial Mixup Regularization

TL;DR

CAARMA tackles limited class diversity in zero-shot speaker verification by generating synthetic speaker identities via embedding-space SL-Mixup and refining them with an adversarial mixup discriminator. The approach expands effective training classes without extra real data and preserves inference cost, yielding up to an ~8% improvement over baselines in SV and gains in emotion and distance tasks. Key contributions include the SL-Mixup mechanism, a HuBERT-based mixup discriminator, and a scalable training framework validated on VoxCeleb1/2 and Dynamic-SUPERB. The work demonstrates robust generalization across tasks and provides public code for reproducibility and broader adoption in zero-shot speech analysis.

Abstract

Speaker verification is a typical zero-shot learning task, where inference of unseen classes is performed by comparing embeddings of test instances to known examples. The models performing inference must hence naturally generate embeddings that cluster same-class instances compactly, while maintaining separation across classes. In order to learn to do so, they are typically trained on a large number of classes (speakers), often using specialized losses. However real-world speaker datasets often lack the class diversity needed to effectively learn this in a generalizable manner. We introduce CAARMA, a class augmentation framework that addresses this problem by generating synthetic classes through data mixing in the embedding space, expanding the number of training classes. To ensure the authenticity of the synthetic classes we adopt a novel adversarial refinement mechanism that minimizes categorical distinctions between synthetic and real classes. We evaluate CAARMA on multiple speaker verification tasks, as well as other representative zero-shot comparison-based speech analysis tasks and obtain consistent improvements: our framework demonstrates a significant improvement of 8\% over all baseline models. The code is available at: https://github.com/massabaali7/CAARMA/

Paper Structure

This paper contains 25 sections, 1 equation, 2 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related-Work
  3. Mixup.
  4. Synthetic speech.
  5. Class Augmentation with Adversarial Mixup Regularization
  6. Overview
  7. Framework
  8. Encoder
  9. Synthetic Label Mixup
  10. Adversarial Training
  11. Mixup Discriminator
  12. Experiments
  13. Datasets
  14. Experimental Setup
  15. Model Architecture
  16. ...and 10 more sections

Figures (2)

  • Figure 1: (a) When trained with fewer classes the model can spread the embeddings of individual classes out while still learning to classify the training data accurately and with large margins. This will not, however translate to compact representations for newer unseen classes. (b) With additional synthetic classes (shaded grey), the model must now learn to compact classes more. This will translate to more compact unseen classes as well.
  • Figure 2: Illustration of CAARMA framework. (a) The encoder ($\mathcal{E}$) extracts embeddings from Mel-spectrograms, which are processed by a classification head ($\mathcal{H}$) for speaker identification and through Mix-Up for synthetic embedding generation. (b) Both real and synthetic embeddings are fed into a pretrained SSL model that acts as a discriminator, distinguishing between real and synthetic samples.