Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language-Universal Speech Attributes Modeling for Zero-Shot Multilingual Spoken Keyword Recognition

Hao Yen, Pin-Jui Ku, Sabato Marco Siniscalchi, Chin-Hui Lee

TL;DR

This work tackles zero-shot multilingual spoken keyword recognition (SKR) by replacing language-specific tokens with universal speech attributes and leveraging a pre-trained self-supervised encoder (Wav2Vec2.0). The system uses a linear output layer to predict attribute posteriors and a non-trainable pronunciation model to map attribute sequences to keywords, with domain adversarial training (DAT) to enforce language-invariant representations via a gradient reversal layer. Empirical results on the Multilingual Spoken Words Corpus show that attribute-based SKR with DAT achieves competitive performance in seen languages and substantial gains in zero-shot scenarios, including improvements of up to around 32% relative WER reduction for unseen languages. The approach demonstrates the practicality and scalability of language-universal SKR and offers a flexible path toward a unified multilingual ASR framework that can generalize to out-of-vocabulary keywords and new languages.

Abstract

We propose a novel language-universal approach to end-to-end automatic spoken keyword recognition (SKR) leveraging upon (i) a self-supervised pre-trained model, and (ii) a set of universal speech attributes (manner and place of articulation). Specifically, Wav2Vec2.0 is used to generate robust speech representations, followed by a linear output layer to produce attribute sequences. A non-trainable pronunciation model then maps sequences of attributes into spoken keywords in a multilingual setting. Experiments on the Multilingual Spoken Words Corpus show comparable performances to character- and phoneme-based SKR in seen languages. The inclusion of domain adversarial training (DAT) improves the proposed framework, outperforming both character- and phoneme-based SKR approaches with 13.73% and 17.22% relative word error rate (WER) reduction in seen languages, and achieves 32.14% and 19.92% WER reduction for unseen languages in zero-shot settings.

Language-Universal Speech Attributes Modeling for Zero-Shot Multilingual Spoken Keyword Recognition

TL;DR

This work tackles zero-shot multilingual spoken keyword recognition (SKR) by replacing language-specific tokens with universal speech attributes and leveraging a pre-trained self-supervised encoder (Wav2Vec2.0). The system uses a linear output layer to predict attribute posteriors and a non-trainable pronunciation model to map attribute sequences to keywords, with domain adversarial training (DAT) to enforce language-invariant representations via a gradient reversal layer. Empirical results on the Multilingual Spoken Words Corpus show that attribute-based SKR with DAT achieves competitive performance in seen languages and substantial gains in zero-shot scenarios, including improvements of up to around 32% relative WER reduction for unseen languages. The approach demonstrates the practicality and scalability of language-universal SKR and offers a flexible path toward a unified multilingual ASR framework that can generalize to out-of-vocabulary keywords and new languages.

Abstract

We propose a novel language-universal approach to end-to-end automatic spoken keyword recognition (SKR) leveraging upon (i) a self-supervised pre-trained model, and (ii) a set of universal speech attributes (manner and place of articulation). Specifically, Wav2Vec2.0 is used to generate robust speech representations, followed by a linear output layer to produce attribute sequences. A non-trainable pronunciation model then maps sequences of attributes into spoken keywords in a multilingual setting. Experiments on the Multilingual Spoken Words Corpus show comparable performances to character- and phoneme-based SKR in seen languages. The inclusion of domain adversarial training (DAT) improves the proposed framework, outperforming both character- and phoneme-based SKR approaches with 13.73% and 17.22% relative word error rate (WER) reduction in seen languages, and achieves 32.14% and 19.92% WER reduction for unseen languages in zero-shot settings.
Paper Structure (17 sections, 1 equation, 1 figure, 6 tables)

This paper contains 17 sections, 1 equation, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language-Universal Multilingual Modeling
  4. Wav2Vec2.0
  5. Domain Adversarial Training (DAT)
  6. Proposed Language-Universal Spoken Keyword Recognition
  7. ASR-based Spoken Keyword Recognition System
  8. Universal Speech Attribute Modeling
  9. DAT for Universal Spoken Keyword Recognition
  10. Experiments and Results
  11. Datasets
  12. Experimental Settings
  13. DAT Results: Language Identification
  14. Recognition Results
  15. In-domain Results: Seen Languages
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Frameworks studied in this work. "W2V2" refers to the Wav2Vec2.0 pre-trained model, and "PM" refers to pronunciation model. In (a), the Baseline system is trained from scratch with different tokens, namely characters, phonemes and speech attributes. In (b), a domain adversarial training (DAT) framework. GRL refers to gradient reversal layer and Classifier denotes a language classifier.