Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models based ASR Error Correction for Child Conversations

Anfeng Xu, Tiantian Feng, So Hyun Kim, Somer Bishop, Catherine Lord, Shrikanth Narayanan

TL;DR

This study tackles ASR errors in child conversational speech by introducing LLM-based error correction and adapting a HyPoradise-style prompting framework to child-adult conversations. It systematically compares zero-shot and fine-tuned ASR outputs across two datasets (MyST and ADOS-Mod3) using prompt designs with and without context, and with instruction-tuned LLMs. Key findings show that large LLMs improve zero-shot corrections for multiple Whisper variants, while improvements for fine-tuned ASR outputs depend on the ASR architecture, with pronounced gains for WavLM-L and limited gains for Whisper; adding conversational context generally degrades performance due to error propagation. The work provides practical insights into the limits of context-driven LLM corrections in child speech and guides future directions toward larger models and improved context integration strategies.

Abstract

Automatic Speech Recognition (ASR) has recently shown remarkable progress, but accurately transcribing children's speech remains a significant challenge. Recent developments in Large Language Models (LLMs) have shown promise in improving ASR transcriptions. However, their applications in child speech including conversational scenarios are underexplored. In this study, we explore the use of LLMs in correcting ASR errors for conversational child speech. We demonstrate the promises and challenges of LLMs through experiments on two children's conversational speech datasets with both zero-shot and fine-tuned ASR outputs. We find that while LLMs are helpful in correcting zero-shot ASR outputs and fine-tuned CTC-based ASR outputs, it remains challenging for LLMs to improve ASR performance when incorporating contextual information or when using fine-tuned autoregressive ASR (e.g., Whisper) outputs.

Large Language Models based ASR Error Correction for Child Conversations

TL;DR

This study tackles ASR errors in child conversational speech by introducing LLM-based error correction and adapting a HyPoradise-style prompting framework to child-adult conversations. It systematically compares zero-shot and fine-tuned ASR outputs across two datasets (MyST and ADOS-Mod3) using prompt designs with and without context, and with instruction-tuned LLMs. Key findings show that large LLMs improve zero-shot corrections for multiple Whisper variants, while improvements for fine-tuned ASR outputs depend on the ASR architecture, with pronounced gains for WavLM-L and limited gains for Whisper; adding conversational context generally degrades performance due to error propagation. The work provides practical insights into the limits of context-driven LLM corrections in child speech and guides future directions toward larger models and improved context integration strategies.

Abstract

Automatic Speech Recognition (ASR) has recently shown remarkable progress, but accurately transcribing children's speech remains a significant challenge. Recent developments in Large Language Models (LLMs) have shown promise in improving ASR transcriptions. However, their applications in child speech including conversational scenarios are underexplored. In this study, we explore the use of LLMs in correcting ASR errors for conversational child speech. We demonstrate the promises and challenges of LLMs through experiments on two children's conversational speech datasets with both zero-shot and fine-tuned ASR outputs. We find that while LLMs are helpful in correcting zero-shot ASR outputs and fine-tuned CTC-based ASR outputs, it remains challenging for LLMs to improve ASR performance when incorporating contextual information or when using fine-tuned autoregressive ASR (e.g., Whisper) outputs.

Paper Structure

This paper contains 19 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. LLM Prompt Design
  4. ASR Error Correction without Context
  5. ASR Error Correction with Context
  6. ASR Models
  7. ASR Fine-tuning
  8. Experiments
  9. Evaluation
  10. Dataset
  11. Experimental Setup: ASR Fine-tuning
  12. Experimental Setup: LLM Instruction tuning
  13. Results and Discussion
  14. Can LLMs Improve zero-shot Child ASR Results?
  15. Can LLMs Improve Fine-tuned Child ASR Results?
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Overall pipeline for ASR with LLM error correction.
  • Figure 2: LLM prompt without context.
  • Figure 3: LLM prompt with context.
  • Figure 4: WERs by utterance lengths with zero-shot Whisper ASR (WSP-L-T). Results from both datasets.
  • Figure 5: WERs by utterance lengths with fine-tuned ASR models (WSP-L-T, WavLM-L), using the ADOS-Mod3 dataset.