Boosting ASR Robustness via Test-Time Reinforcement Learning with Audio-Text Semantic Rewards

TL;DR

This work presents ASR-TRA, a novel Test-time Reinforcement Adaptation framework inspired by causal intervention, which introduces a learnable decoder prompt and utilizes temperature-controlled stochastic decoding to generate diverse transcription candidates.

Abstract

Recently, Automatic Speech Recognition (ASR) systems (e.g., Whisper) have achieved remarkable accuracy improvements but remain highly sensitive to real-world unseen data (data with large distribution shifts), including noisy environments and diverse accents. To address this issue, test-time adaptation (TTA) has shown great potential in improving the model adaptability at inference time without ground-truth labels, and existing TTA methods often rely on pseudo-labeling or entropy minimization. However, by treating model confidence as a learning signal, these methods may reinforce high-confidence errors, leading to confirmation bias that undermines adaptation. To overcome these limitations, we present ASR-TRA, a novel Test-time Reinforcement Adaptation framework inspired by causal intervention. More precisely, our method introduces a learnable decoder prompt and utilizes temperature-controlled stochastic decoding to generate diverse transcription candidates. These are scored by a reward model that measures audio-text semantic alignment, and the resulting feedback is used to update both model and prompt parameters via reinforcement learning. Comprehensive experiments on LibriSpeech with synthetic noise and L2 Arctic accented English datasets demonstrate that our method achieves higher accuracy while maintaining lower latency than existing TTA baselines. Ablation studies further confirm the effectiveness of combining audio and language-based rewards, highlighting our method's enhanced stability and interpretability. Overall, our approach provides a practical and robust solution for deploying ASR systems in challenging real-world conditions.

Figures (4)

• Figure 1: Overview of test-time adaptation strategies under noisy conditions. (a) Confidence-driven adaptation reinforces incorrect but high-probability predictions (e.g., word), leading to persistent errors. (b) Our reward-guided adaptation ASR-TRA favors semantically accurate alternatives (e.g., world) through reinforcement learning, even when their initial confidence is low.
• Figure 2: We evaluate WER on the top-100 high-confidence samples from LibriSpeech test-other corrupted with Gaussian noise. While baseline performance from Whisper-Tiny suffers under noise, heuristic methods like SUTA further degrade due to overconfidence. In contrast, ours achieves lower WER. (Lower is better.)
• Figure 3: Structural Causal Model (SCM) schematic. Nodes $A$, $P$, $Y$, and $R$ denote audio features, learnable prompt, transcription output, and reward, respectively, with causal flow $A,P \!\rightarrow\! Y \!\rightarrow\! R$pearl2009causality.
• Figure 4: Test‑time self‑adaptation for Whisper. A baseline transcript is decoded from the input Mel‑spectrogram; soft‑prompted variants are then sampled at varied temperatures, CLAP scores them all, and the aggregated rewards update the model, achieving on‑the‑fly correction without labels.