Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognition

Yuu Jinnai

TL;DR

This paper investigates the effectiveness of sample-based Minimum Bayes Risk (MBR) decoding for offline automatic speech recognition and speech translation, across English and Japanese using Whisper-based models. It systematically compares MBR against beam search under diverse datasets, languages, and noisy conditions, using unbiased sampling with a BLEU-based utility and evaluating with WER/CER, SemDist, and MetricX. The study demonstrates that MBR consistently yields higher accuracy than beam search in most settings, with theoretical guarantees of improved expected utility as the sample size grows, and finds that modest sample sizes (4–8) already provide meaningful gains. However, the approach incurs substantial computational cost and may not suit real-time deployment; nonetheless, the results establish MBR as a strong, training-free baseline for high-accuracy offline ASR and ST, with code released for reproducibility. The work highlights MBR’s robustness across languages and noise levels and its potential to complement or surpass reranking methods in contemporary speech systems.

Abstract

Recent work has shown that sample-based Minimum Bayes Risk (MBR) decoding outperforms beam search in text-to-text generation tasks, such as machine translation, text summarization, and image captioning. On the other hand, beam search is the current practice for speech-to-text tasks such as automatic speech recognition (ASR) and Speech Translation (ST). Given that MBR decoding is effective in text-to-text generation tasks, it is reasonable to expect it to also be effective for speech-to-text tasks. In this paper, we evaluate MBR decoding for ASR and ST tasks on English and Japanese using Whisper and its derivative models. We observe that the accuracy of MBR decoding outperforms that of beam search in most of the experimental settings we have evaluated. The results show that MBR decoding is a promising method for offline ASR and ST tasks that require high accuracy. The code is available at https://github.com/CyberAgentAILab/mbr-for-asr

Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognition

TL;DR

This paper investigates the effectiveness of sample-based Minimum Bayes Risk (MBR) decoding for offline automatic speech recognition and speech translation, across English and Japanese using Whisper-based models. It systematically compares MBR against beam search under diverse datasets, languages, and noisy conditions, using unbiased sampling with a BLEU-based utility and evaluating with WER/CER, SemDist, and MetricX. The study demonstrates that MBR consistently yields higher accuracy than beam search in most settings, with theoretical guarantees of improved expected utility as the sample size grows, and finds that modest sample sizes (4–8) already provide meaningful gains. However, the approach incurs substantial computational cost and may not suit real-time deployment; nonetheless, the results establish MBR as a strong, training-free baseline for high-accuracy offline ASR and ST, with code released for reproducibility. The work highlights MBR’s robustness across languages and noise levels and its potential to complement or surpass reranking methods in contemporary speech systems.

Abstract

Recent work has shown that sample-based Minimum Bayes Risk (MBR) decoding outperforms beam search in text-to-text generation tasks, such as machine translation, text summarization, and image captioning. On the other hand, beam search is the current practice for speech-to-text tasks such as automatic speech recognition (ASR) and Speech Translation (ST). Given that MBR decoding is effective in text-to-text generation tasks, it is reasonable to expect it to also be effective for speech-to-text tasks. In this paper, we evaluate MBR decoding for ASR and ST tasks on English and Japanese using Whisper and its derivative models. We observe that the accuracy of MBR decoding outperforms that of beam search in most of the experimental settings we have evaluated. The results show that MBR decoding is a promising method for offline ASR and ST tasks that require high accuracy. The code is available at https://github.com/CyberAgentAILab/mbr-for-asr
Paper Structure (25 sections, 3 equations, 3 figures, 10 tables)

This paper contains 25 sections, 3 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Text Generation Problem
  4. Minimum Bayes Risk (MBR) Decoding
  5. Related Work
  6. Experiments
  7. Method.
  8. Implementation.
  9. Automatic Speech Recognition (ASR)
  10. Resources.
  11. Evaluation metrics.
  12. Evaluation metrics.
  13. Models.
  14. Number of samples for MBR decoding.
  15. Correlation of MBR objective values to error rates.
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Illustration of the beam search and MBR decoding: multiple hypotheses are sampled from the ASR model, and the hypothesis with the highest expected utility (e.g., BLEU score) compared to the others is selected as the final output.
  • Figure 2: Illustrative explanation of the intuition of the MBR decoding. The hypothesis that lies at the center of the sampled hypotheses is selected as the output. The distance between two hypotheses is inversely related to their utility.
  • Figure 3: WER of AMI-IHM averaged over the instances with the number of words in the reference text is in the range of (x, x+5].