Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks

Kevin Everson, Yile Gu, Huck Yang, Prashanth Gurunath Shivakumar, Guan-Ting Lin, Jari Kolehmainen, Ivan Bulyko, Ankur Gandhe, Shalini Ghosh, Wael Hamza, Hung-yi Lee, Ariya Rastrow, Andreas Stolcke

TL;DR

This work addresses SLU robustness under ASR errors by representing ASR output as word confusion networks (WCNs) derived from lattices and feeding these representations into off-the-shelf LLMs via prompting, without fine-tuning. The study demonstrates that WCNs can bridge part of the gap between top-ASR hypotheses and oracle performance for both spoken question answering and intent classification, with notable gains when using GPT-3.5-turbo and appropriate prompting (including WCN instruction and posterior filtering). Key findings show model size and in-context learning design critically influence gains, with WCN-based prompts providing resilience across a range of ASR qualities, though benefits are limited for smaller models. Overall, the approach offers a simple, non-tuning method to enhance SLU performance in real-world, noisy ASR conditions, highlighting practical implications for deployed voice-enabled systems.

Abstract

In the realm of spoken language understanding (SLU), numerous natural language understanding (NLU) methodologies have been adapted by supplying large language models (LLMs) with transcribed speech instead of conventional written text. In real-world scenarios, prior to input into an LLM, an automated speech recognition (ASR) system generates an output transcript hypothesis, where inherent errors can degrade subsequent SLU tasks. Here we introduce a method that utilizes the ASR system's lattice output instead of relying solely on the top hypothesis, aiming to encapsulate speech ambiguities and enhance SLU outcomes. Our in-context learning experiments, covering spoken question answering and intent classification, underline the LLM's resilience to noisy speech transcripts with the help of word confusion networks from lattices, bridging the SLU performance gap between using the top ASR hypothesis and an oracle upper bound. Additionally, we delve into the LLM's robustness to varying ASR performance conditions and scrutinize the aspects of in-context learning which prove the most influential.

Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks

TL;DR

This work addresses SLU robustness under ASR errors by representing ASR output as word confusion networks (WCNs) derived from lattices and feeding these representations into off-the-shelf LLMs via prompting, without fine-tuning. The study demonstrates that WCNs can bridge part of the gap between top-ASR hypotheses and oracle performance for both spoken question answering and intent classification, with notable gains when using GPT-3.5-turbo and appropriate prompting (including WCN instruction and posterior filtering). Key findings show model size and in-context learning design critically influence gains, with WCN-based prompts providing resilience across a range of ASR qualities, though benefits are limited for smaller models. Overall, the approach offers a simple, non-tuning method to enhance SLU performance in real-world, noisy ASR conditions, highlighting practical implications for deployed voice-enabled systems.

Abstract

In the realm of spoken language understanding (SLU), numerous natural language understanding (NLU) methodologies have been adapted by supplying large language models (LLMs) with transcribed speech instead of conventional written text. In real-world scenarios, prior to input into an LLM, an automated speech recognition (ASR) system generates an output transcript hypothesis, where inherent errors can degrade subsequent SLU tasks. Here we introduce a method that utilizes the ASR system's lattice output instead of relying solely on the top hypothesis, aiming to encapsulate speech ambiguities and enhance SLU outcomes. Our in-context learning experiments, covering spoken question answering and intent classification, underline the LLM's resilience to noisy speech transcripts with the help of word confusion networks from lattices, bridging the SLU performance gap between using the top ASR hypothesis and an oracle upper bound. Additionally, we delve into the LLM's robustness to varying ASR performance conditions and scrutinize the aspects of in-context learning which prove the most influential.
Paper Structure (19 sections, 3 figures, 7 tables)

This paper contains 19 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Word confusion networks (WCN) input representation
  4. ASR system and WCN construction
  5. LLM inputs from WCNs
  6. In-context learning for SLU with WCNs
  7. Language models
  8. Baselines
  9. A Study of Spoken Question Answering
  10. NMSQA dataset and task
  11. Prompt design and spoken QA results
  12. A Study of Intent Classification
  13. ATIS dataset and task
  14. Prompt design and results
  15. Analysis and Discussion
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Proposed in-context learning framework using a WCN question transcript for Spoken Language Understanding (SLU).
  • Figure 2: Constructing LLM input from raw WCN: (a) Raw WCN, (b) LLM input from WCN, and (c) LLM input from WCN after filtering based on word posteriors ($p \geq 0.3$).
  • Figure 3: NMSQA test-set one-shot EM by question WER bin.