Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interventional Speech Noise Injection for ASR Generalizable Spoken Language Understanding

Yeonjoon Jung, Jaeseong Lee, Seungtaek Choi, Dohyeon Lee, Minsoo Kim, Seung-won Hwang

TL;DR

The effectiveness of the proposed methods in enhancing the robustness and generalizability of SLU models against unseen ASR systems by introducing more diverse and plausible ASR noises in advance is demonstrated.

Abstract

Recently, pre-trained language models (PLMs) have been increasingly adopted in spoken language understanding (SLU). However, automatic speech recognition (ASR) systems frequently produce inaccurate transcriptions, leading to noisy inputs for SLU models, which can significantly degrade their performance. To address this, our objective is to train SLU models to withstand ASR errors by exposing them to noises commonly observed in ASR systems, referred to as ASR-plausible noises. Speech noise injection (SNI) methods have pursued this objective by introducing ASR-plausible noises, but we argue that these methods are inherently biased towards specific ASR systems, or ASR-specific noises. In this work, we propose a novel and less biased augmentation method of introducing the noises that are plausible to any ASR system, by cutting off the non-causal effect of noises. Experimental results and analyses demonstrate the effectiveness of our proposed methods in enhancing the robustness and generalizability of SLU models against unseen ASR systems by introducing more diverse and plausible ASR noises in advance.

Interventional Speech Noise Injection for ASR Generalizable Spoken Language Understanding

TL;DR

The effectiveness of the proposed methods in enhancing the robustness and generalizability of SLU models against unseen ASR systems by introducing more diverse and plausible ASR noises in advance is demonstrated.

Abstract

Recently, pre-trained language models (PLMs) have been increasingly adopted in spoken language understanding (SLU). However, automatic speech recognition (ASR) systems frequently produce inaccurate transcriptions, leading to noisy inputs for SLU models, which can significantly degrade their performance. To address this, our objective is to train SLU models to withstand ASR errors by exposing them to noises commonly observed in ASR systems, referred to as ASR-plausible noises. Speech noise injection (SNI) methods have pursued this objective by introducing ASR-plausible noises, but we argue that these methods are inherently biased towards specific ASR systems, or ASR-specific noises. In this work, we propose a novel and less biased augmentation method of introducing the noises that are plausible to any ASR system, by cutting off the non-causal effect of noises. Experimental results and analyses demonstrate the effectiveness of our proposed methods in enhancing the robustness and generalizability of SLU models against unseen ASR systems by introducing more diverse and plausible ASR noises in advance.

Paper Structure

This paper contains 39 sections, 18 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. SNI
  4. ASR Correction
  5. Method
  6. Problem Formulation
  7. Causality in SNI
  8. $do$-calculus for ISNI
  9. Overview of Our Proposed Approach
  10. Interventional SNI for Debiasing
  11. Phoneme-aware Generation for Generalizability
  12. Experiments
  13. Dataset
  14. SNI Training
  15. SLU Training
  16. ...and 24 more sections

Figures (3)

  • Figure 1: Different ASR systems generate different ASR errors (ASR$_1$ : blue, ASR$_2$ : red, common, or, ASR$_*$ : green). Biased toward a specific ASR, ASR$_1$, baseline SNI generates noises plausible only for ASR$_1$, or even some noise that are not plausible to any (cue to sue). Our distinction is 1) removing its bias to a specific ASR (Read to Lead), and 2) generating ASR$_*$-plausible noises (cue to queue).
  • Figure 2: The causal graph between ASR transcription (\ref{['fig:fig2_subfig1']}), SNI training data generation (\ref{['fig:fig2_subfig2']}), and ISNI (\ref{['fig:fig2_subfig3']})
  • Figure 3: Overview of ISNI. ISNI generates ASR noise word $t^k$ for the clean text $x^k$ whose corresponding $z^k$ is 1. The error type of $t^k$ is determined by the generated output.