Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projection

Sergio Burdisso, Esaú Villatoro-Tello, Shashi Kumar, Srikanth Madikeri, Andrés Carofilis, Pradeep Rangappa, Manjunath K E, Kadri Hacioglu, Petr Motlicek, Andreas Stolcke

TL;DR

This paper addresses prompt sensitivity in LLM-based ASR by showing that fixed prompts cause substantial performance variability across datasets. It introduces a learnable prompt projector that maps prompt embeddings into more effective regions of the LLM input space while keeping the LLM and speech encoder unchanged. Across multiple datasets, the prompt projector reduces prompt-induced variability and closes the gap to the best manual prompts, often outperforming them. The authors provide code to facilitate reproduction and discuss future work comparing with soft-prompt approaches and extending the method to other models and tasks.

Abstract

LLM-based automatic speech recognition (ASR), a well-established approach, connects speech foundation models to large language models (LLMs) through a speech-to-LLM projector, yielding promising results. A common design choice in these architectures is the use of a fixed, manually defined prompt during both training and inference. This setup not only enables applicability across a range of practical scenarios, but also helps maximize model performance. However, the impact of prompt design remains underexplored. This paper presents a comprehensive analysis of commonly used prompts across diverse datasets, showing that prompt choice significantly affects ASR performance and introduces instability, with no single prompt performing best across all cases. Inspired by the speech-to-LLM projector, we propose a prompt projector module, a simple, model-agnostic extension that learns to project prompt embeddings to more effective regions of the LLM input space, without modifying the underlying LLM-based ASR model. Experiments on four datasets show that the addition of a prompt projector consistently improves performance, reduces variability, and outperforms the best manually selected prompts.

Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projection

TL;DR

This paper addresses prompt sensitivity in LLM-based ASR by showing that fixed prompts cause substantial performance variability across datasets. It introduces a learnable prompt projector that maps prompt embeddings into more effective regions of the LLM input space while keeping the LLM and speech encoder unchanged. Across multiple datasets, the prompt projector reduces prompt-induced variability and closes the gap to the best manual prompts, often outperforming them. The authors provide code to facilitate reproduction and discuss future work comparing with soft-prompt approaches and extending the method to other models and tasks.

Abstract

LLM-based automatic speech recognition (ASR), a well-established approach, connects speech foundation models to large language models (LLMs) through a speech-to-LLM projector, yielding promising results. A common design choice in these architectures is the use of a fixed, manually defined prompt during both training and inference. This setup not only enables applicability across a range of practical scenarios, but also helps maximize model performance. However, the impact of prompt design remains underexplored. This paper presents a comprehensive analysis of commonly used prompts across diverse datasets, showing that prompt choice significantly affects ASR performance and introduces instability, with no single prompt performing best across all cases. Inspired by the speech-to-LLM projector, we propose a prompt projector module, a simple, model-agnostic extension that learns to project prompt embeddings to more effective regions of the LLM input space, without modifying the underlying LLM-based ASR model. Experiments on four datasets show that the addition of a prompt projector consistently improves performance, reduces variability, and outperforms the best manually selected prompts.
Paper Structure (10 sections, 1 equation, 3 figures, 3 tables)

This paper contains 10 sections, 1 equation, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Base Model
  4. The Prompt Projector: $pp(\cdot)$
  5. Training & Inference Implementation Details
  6. Datasets
  7. Experimentation
  8. Assessing the Impact of Manual Prompts
  9. Incorporating the Prompt Projector
  10. Conclusions

Figures (3)

  • Figure 1: Typical LLM-based ASR system composed of a fixed prompt, a speech encoder, and an LLM connected by a speech projector, $sp(\cdot)$. The proposed extension, highlighted in orange, introduces a learnable prompt projector, $pp(\cdot)$, into the original (frozen) architecture. The $pp(\cdot)$ learns a common/single projection to transform all the original prompt embeddings ($\blacktriangle$) into more effective ones ($\blacktriangle$).
  • Figure 2: ASR performance (WER (in %)) across datasets with different prompts. The black dashed line represents the base prompt, while the blue dotted line corresponds to the empty prompt.
  • Figure 3: Boxplots illustrating the impact of applying $pp(\cdot)$ across different datasets. Each subplot compares WER (in %) distributions among the different prompts before (vanilla) and after applying $pp(\cdot)$. Improvements are statistically significant across all datasets according to paired statistical tests ($p<0.05$).