Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties

Nathan Roll, Calbert Graham, Yuka Tatsumi, Kim Tien Nguyen, Meghan Sumner, Dan Jurafsky

TL;DR

The paper addresses how to achieve human-like adaptation in state-of-the-art spoken language models via in-context learning for ASR. It introduces a scalable, prompt-based ICL framework for Phi-4-MM that interleaves audio-text exemplars with task prompts, enabling rapid adaptation at inference without fine-tuning. Across four English corpora with native, heritage, and non-native speakers, the method delivers a mean WER reduction of about 19.7% relative when using 9–12 shots, with the largest gains for low-resource varieties and a two-phase adaptation pattern (speaker-specific then variety-general). The findings suggest ICL can improve ASR robustness and equity across diverse speakers and languages, while also highlighting remaining gaps and the importance of prompt design; prompts and code are released for reproducibility.

Abstract

Human listeners readily adjust to unfamiliar speakers and language varieties through exposure, but do these adaptation benefits extend to state-of-the-art spoken language models? We introduce a scalable framework that allows for in-context learning (ICL) in Phi-4 Multimodal using interleaved task prompts and audio-text pairs, and find that as few as 12 example utterances (~50 seconds) at inference time reduce word error rates by a relative 19.7% (1.2 pp.) on average across diverse English corpora. These improvements are most pronounced in low-resource varieties, when the context and target speaker match, and when more examples are provided--though scaling our procedure yields diminishing marginal returns to context length. Overall, we find that our novel ICL adaptation scheme (1) reveals a similar performance profile to human listeners, and (2) demonstrates consistent improvements to automatic speech recognition (ASR) robustness across diverse speakers and language backgrounds. While adaptation succeeds broadly, significant gaps remain for certain varieties, revealing where current models still fall short of human flexibility. We release our prompts and code on GitHub.

In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties

TL;DR

The paper addresses how to achieve human-like adaptation in state-of-the-art spoken language models via in-context learning for ASR. It introduces a scalable, prompt-based ICL framework for Phi-4-MM that interleaves audio-text exemplars with task prompts, enabling rapid adaptation at inference without fine-tuning. Across four English corpora with native, heritage, and non-native speakers, the method delivers a mean WER reduction of about 19.7% relative when using 9–12 shots, with the largest gains for low-resource varieties and a two-phase adaptation pattern (speaker-specific then variety-general). The findings suggest ICL can improve ASR robustness and equity across diverse speakers and languages, while also highlighting remaining gaps and the importance of prompt design; prompts and code are released for reproducibility.

Abstract

Human listeners readily adjust to unfamiliar speakers and language varieties through exposure, but do these adaptation benefits extend to state-of-the-art spoken language models? We introduce a scalable framework that allows for in-context learning (ICL) in Phi-4 Multimodal using interleaved task prompts and audio-text pairs, and find that as few as 12 example utterances (~50 seconds) at inference time reduce word error rates by a relative 19.7% (1.2 pp.) on average across diverse English corpora. These improvements are most pronounced in low-resource varieties, when the context and target speaker match, and when more examples are provided--though scaling our procedure yields diminishing marginal returns to context length. Overall, we find that our novel ICL adaptation scheme (1) reveals a similar performance profile to human listeners, and (2) demonstrates consistent improvements to automatic speech recognition (ASR) robustness across diverse speakers and language backgrounds. While adaptation succeeds broadly, significant gaps remain for certain varieties, revealing where current models still fall short of human flexibility. We release our prompts and code on GitHub.

Paper Structure

This paper contains 20 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Data
  4. L2-ARCTIC
  5. CMU-Arctic
  6. Hispanic-English Corpus (HEC)
  7. Speech Accent Archive (SAA)
  8. Data Selection Rationale
  9. Model: Phi-4-MM
  10. Methods
  11. In-Context Learning Framework
  12. Prompt Design and Speaker Context Conditions
  13. Model Configuration and Preprocessing
  14. Experimental Design and Evaluation
  15. Results
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Our framework provides an initial description along with N transcribed examples (blue) before tasking the model to transcribe the final ASR objective audio (red). Phi-4-MM interleaves text (orange) with audio (green). These are projected into a multimodal embedding space, the context window of the shared decoder.
  • Figure 2: In-context learning consistently reduces WERs across all corpora with diminishing returns. (Left) WER trajectories by shot count show rapid initial improvement plateauing around 6-10 examples. (Right) Aggregated results across shot buckets (0-3, 4-8, 9-12) demonstrate strictly decreasing WERs with more examples. CMU-Arctic (native speakers) achieves lowest WERs across all conditions.
  • Figure 3: Prompt format sensitivity across corpora. Including explicit "Transcription:" markers reduces WERs in low-resource corpora (HEC, L2-Arctic). In zero shot settings, marginal gains are induced simply by informing the model that it's transcribing non-native speech.