Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RLBR: Reinforcement Learning with Biasing Rewards for Contextual Speech Large Language Models

Bo Ren, Ruchao Fan, Yelong Shen, Weizhu Chen, Jinyu Li

TL;DR

The paper tackles contextual biasing in speech LLMs, where rare and domain-specific terms are often misrecognized. It introduces RLBR, a reinforcement learning fine-tuning method built on GRPO that explicitly prioritizes biasing words through a biasing-reward, and enhances exploration with a reference-aware trajectory mechanism. Key innovations include a biasing word–weighted reward $ r_i= - ( \mathcal{ED}(o^{*}, o_i) + \lambda \mathcal{ED}_{b}(o^{*}, o_i) )$ and a reference-aware GRPO that includes the ground-truth transcription as an additional hypothesis, expanding the reward set $R^{+}$. Experiments on LibriSpeech across biasing list sizes show substantial BWER reductions (e.g., $0.59\%/2.11\%$, $1.09\%/3.24\%$, $1.36\%/4.04\%$) with Lambda set to $5$, while maintaining overall WER and UWER, demonstrating robust contextual adaptation without changing decoding or architecture. The work suggests RL-based contextual biasing can generalize to other speech tasks and domain-specific applications, providing a practical pathway to more accurate transcription of rare terms.

Abstract

Speech large language models (LLMs) have driven significant progress in end-to-end speech understanding and recognition, yet they continue to struggle with accurately recognizing rare words and domain-specific terminology. This paper presents a novel fine-tuning method, Reinforcement Learning with Biasing Rewards (RLBR), which employs a specialized biasing words preferred reward to explicitly emphasize biasing words in the reward calculation. In addition, we introduce reference-aware mechanisms that extend the reinforcement learning algorithm with reference transcription to strengthen the potential trajectory exploration space. Experiments on the LibriSpeech corpus across various biasing list sizes demonstrate that RLBR delivers substantial performance improvements over a strong supervised fine-tuning (SFT) baseline and consistently outperforms several recently published methods. The proposed approach achieves excellent performance on the LibriSpeech test-clean and test-other sets, reaching Biasing Word Error Rates (BWERs) of 0.59% / 2.11%, 1.09% / 3.24%, and 1.36% / 4.04% for biasing list sizes of 100, 500, and 1000, respectively, without compromising the overall WERs.

RLBR: Reinforcement Learning with Biasing Rewards for Contextual Speech Large Language Models

TL;DR

The paper tackles contextual biasing in speech LLMs, where rare and domain-specific terms are often misrecognized. It introduces RLBR, a reinforcement learning fine-tuning method built on GRPO that explicitly prioritizes biasing words through a biasing-reward, and enhances exploration with a reference-aware trajectory mechanism. Key innovations include a biasing word–weighted reward and a reference-aware GRPO that includes the ground-truth transcription as an additional hypothesis, expanding the reward set . Experiments on LibriSpeech across biasing list sizes show substantial BWER reductions (e.g., , , ) with Lambda set to , while maintaining overall WER and UWER, demonstrating robust contextual adaptation without changing decoding or architecture. The work suggests RL-based contextual biasing can generalize to other speech tasks and domain-specific applications, providing a practical pathway to more accurate transcription of rare terms.

Abstract

Speech large language models (LLMs) have driven significant progress in end-to-end speech understanding and recognition, yet they continue to struggle with accurately recognizing rare words and domain-specific terminology. This paper presents a novel fine-tuning method, Reinforcement Learning with Biasing Rewards (RLBR), which employs a specialized biasing words preferred reward to explicitly emphasize biasing words in the reward calculation. In addition, we introduce reference-aware mechanisms that extend the reinforcement learning algorithm with reference transcription to strengthen the potential trajectory exploration space. Experiments on the LibriSpeech corpus across various biasing list sizes demonstrate that RLBR delivers substantial performance improvements over a strong supervised fine-tuning (SFT) baseline and consistently outperforms several recently published methods. The proposed approach achieves excellent performance on the LibriSpeech test-clean and test-other sets, reaching Biasing Word Error Rates (BWERs) of 0.59% / 2.11%, 1.09% / 3.24%, and 1.36% / 4.04% for biasing list sizes of 100, 500, and 1000, respectively, without compromising the overall WERs.
Paper Structure (18 sections, 4 equations, 2 figures, 3 tables)

This paper contains 18 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Group Relative Policy Optimization
  4. GRPO for Contextual biasing
  5. Reinforcement Learning with Biasing Rewards
  6. Reference-aware GRPO
  7. Experimental Setup
  8. Datasets
  9. Contextual data augmentation
  10. Implementation Details
  11. Results and Discussion
  12. Main Results
  13. Ablation Study on RLBR
  14. Biasing Weight in Reward
  15. Biasing Words Formatting Impact
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Illustration of the biasing word preferred reward rule. Word-level edit distance is shown: $\textcolor{green}{\mathcal{ED}(o^{*}, o_{i})}=3$ for all words, and $\textcolor{red}{\mathcal{ED}_{b}(o^{*}, o_{i})}=2$ for biasing words only.
  • Figure 2: Reference-aware GRPO for Speech LLMs.