Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial synthesis based data-augmentation for code-switched spoken language identification

Parth Shastri, Chirag Patil, Poorval Wanere, Shrinivas Mahajan, Abhishek Bhatt, Hardik Sailor

TL;DR

This work tackles code-switched Hindi-English spoken language identification in data-scarce settings by employing a conditional DCGAN that generates 128x128 Mel spectrograms conditioned on the log F0 contour to augment the minority Hindi-English class. The GAN training uses a Wasserstein loss with Gradient Penalty and a reconstruction term, with realism assessed via Frechet Inception Distance. Empirical results on a three-class LID task show up to an 8.1% improvement in F1-score and a 4.7% improvement in UAR over baselines, validating GAN-based representation learning for data imbalance in code-switched speech. The approach demonstrates the potential of synthesis-based augmentation to enhance performance in ASR-related language identification and sets directions for higher-resolution spectrograms and phonetic conditioning.

Abstract

Spoken Language Identification (LID) is an important sub-task of Automatic Speech Recognition(ASR) that is used to classify the language(s) in an audio segment. Automatic LID plays an useful role in multilingual countries. In various countries, identifying a language becomes hard, due to the multilingual scenario where two or more than two languages are mixed together during conversation. Such phenomenon of speech is called as code-mixing or code-switching. This nature is followed not only in India but also in many Asian countries. Such code-mixed data is hard to find, which further reduces the capabilities of the spoken LID. Hence, this work primarily addresses this problem using data augmentation as a solution on the on the data scarcity of the code-switched class. This study focuses on Indic language code-mixed with English. Spoken LID is performed on Hindi, code-mixed with English. This research proposes Generative Adversarial Network (GAN) based data augmentation technique performed using Mel spectrograms for audio data. GANs have already been proven to be accurate in representing the real data distribution in the image domain. Proposed research exploits these capabilities of GANs in speech domains such as speech classification, automatic speech recognition, etc. GANs are trained to generate Mel spectrograms of the minority code-mixed class which are then used to augment data for the classifier. Utilizing GANs give an overall improvement on Unweighted Average Recall by an amount of 3.5% as compared to a Convolutional Recurrent Neural Network (CRNN) classifier used as the baseline reference.

Adversarial synthesis based data-augmentation for code-switched spoken language identification

TL;DR

This work tackles code-switched Hindi-English spoken language identification in data-scarce settings by employing a conditional DCGAN that generates 128x128 Mel spectrograms conditioned on the log F0 contour to augment the minority Hindi-English class. The GAN training uses a Wasserstein loss with Gradient Penalty and a reconstruction term, with realism assessed via Frechet Inception Distance. Empirical results on a three-class LID task show up to an 8.1% improvement in F1-score and a 4.7% improvement in UAR over baselines, validating GAN-based representation learning for data imbalance in code-switched speech. The approach demonstrates the potential of synthesis-based augmentation to enhance performance in ASR-related language identification and sets directions for higher-resolution spectrograms and phonetic conditioning.

Abstract

Spoken Language Identification (LID) is an important sub-task of Automatic Speech Recognition(ASR) that is used to classify the language(s) in an audio segment. Automatic LID plays an useful role in multilingual countries. In various countries, identifying a language becomes hard, due to the multilingual scenario where two or more than two languages are mixed together during conversation. Such phenomenon of speech is called as code-mixing or code-switching. This nature is followed not only in India but also in many Asian countries. Such code-mixed data is hard to find, which further reduces the capabilities of the spoken LID. Hence, this work primarily addresses this problem using data augmentation as a solution on the on the data scarcity of the code-switched class. This study focuses on Indic language code-mixed with English. Spoken LID is performed on Hindi, code-mixed with English. This research proposes Generative Adversarial Network (GAN) based data augmentation technique performed using Mel spectrograms for audio data. GANs have already been proven to be accurate in representing the real data distribution in the image domain. Proposed research exploits these capabilities of GANs in speech domains such as speech classification, automatic speech recognition, etc. GANs are trained to generate Mel spectrograms of the minority code-mixed class which are then used to augment data for the classifier. Utilizing GANs give an overall improvement on Unweighted Average Recall by an amount of 3.5% as compared to a Convolutional Recurrent Neural Network (CRNN) classifier used as the baseline reference.
Paper Structure (20 sections, 14 equations, 8 figures, 6 tables)

This paper contains 20 sections, 14 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. GANs: an Overview
  3. Proposed Method
  4. Motivation
  5. Method Description
  6. Details of implementation
  7. Architecture Details
  8. Dataset for training the GAN
  9. Evaluation of GAN
  10. Experiments
  11. Language identification dataset
  12. Classification and feature extraction
  13. Train-Test data Split
  14. Dataset imbalance
  15. Refernce baselines
  16. ...and 5 more sections

Figures (8)

  • Figure 1: Proposed methodology for spoken LID using CNN-LSTM Network.
  • Figure 2: Architecture of the proposed GAN
  • Figure 3: The FID trend for $n=2400$ with respect to the number of training iterations(steps) of the proposed GAN.
  • Figure 4: Generated images from the proposed GAN. (a) Real Mel spectrograms from the test data. (b) Normalized $\log{F0}$ contours of the audio associated with the real spectrograms. (c) Generated Mel spectrograms by the proposed method.
  • Figure 5: The classifier architecture.
  • ...and 3 more figures