Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Whisper-RIR-Mega: A Paired Clean-Reverberant Speech Benchmark for ASR Robustness to Room Acoustics

Mandip Goswami

TL;DR

The Whisper-RIR-Mega dataset, a benchmark dataset of paired clean and reverberant speech for evaluating automatic speech recognition (ASR) robustness to room acoustics, is introduced and the dataset, evaluation code, and baseline results are released to support reproducible research on robust ASR.

Abstract

We introduce Whisper-RIR-Mega, a benchmark dataset of paired clean and reverberant speech for evaluating automatic speech recognition (ASR) robustness to room acoustics. Each sample pairs a clean LibriSpeech utterance with the same utterance convolved with a real room impulse response from the RIR-Mega corpus, with stratified splits by reverberation time (RT60) and direct-to-reverberant ratio (DRR). We evaluate five Whisper models (tiny through large-v3) on 1600 test samples and report word error rate (WER) and character error rate (CER) under clean and reverberant conditions. Reverberation consistently degrades performance across all model sizes; the reverb penalty in WER ranges from 0.12 to 1.07 percentage points depending on the model. We release the dataset, evaluation code, and baseline results to support reproducible research on robust ASR.

Whisper-RIR-Mega: A Paired Clean-Reverberant Speech Benchmark for ASR Robustness to Room Acoustics

TL;DR

The Whisper-RIR-Mega dataset, a benchmark dataset of paired clean and reverberant speech for evaluating automatic speech recognition (ASR) robustness to room acoustics, is introduced and the dataset, evaluation code, and baseline results are released to support reproducible research on robust ASR.

Abstract

We introduce Whisper-RIR-Mega, a benchmark dataset of paired clean and reverberant speech for evaluating automatic speech recognition (ASR) robustness to room acoustics. Each sample pairs a clean LibriSpeech utterance with the same utterance convolved with a real room impulse response from the RIR-Mega corpus, with stratified splits by reverberation time (RT60) and direct-to-reverberant ratio (DRR). We evaluate five Whisper models (tiny through large-v3) on 1600 test samples and report word error rate (WER) and character error rate (CER) under clean and reverberant conditions. Reverberation consistently degrades performance across all model sizes; the reverb penalty in WER ranges from 0.12 to 1.07 percentage points depending on the model. We release the dataset, evaluation code, and baseline results to support reproducible research on robust ASR.
Paper Structure (12 sections, 1 figure, 1 table)

This paper contains 12 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Construction
  4. Speech and RIR Sources
  5. Signal Processing
  6. Splits
  7. Experimental Setup
  8. Models
  9. Metrics
  10. Results
  11. Discussion
  12. Conclusion

Figures (1)

  • Figure 1: WER by model: clean (blue) and reverberant (coral).