Linear-time Minimum Bayes Risk Decoding with Reference Aggregation

TL;DR

This work tackles the prohibitive quadratic cost of minimum Bayes risk (MBR) decoding in neural machine translation by introducing reference aggregation. By averaging reference representations (e.g., n-gram counts for ChrF or embeddings for COMET) into a single aggregate, utility estimation becomes linear in the number of hypotheses and references, $O(|hyps|+|refs|)$, while preserving most MBR gains. The approach is instantiated for ChrF and COMET, with additional variants like partial aggregation and aggregate-to-fine MBR to balance speed and accuracy. Empirical results across four language directions show dramatic speedups (up to ~99.5% in ChrF and ~95% in COMET) and substantial reductions in end-to-end translation time, bringing MBR closer to practical deployment, albeit with some limitations for trained metrics and potential gains from future improvements in sampling efficiency.

Abstract

Minimum Bayes Risk (MBR) decoding is a text generation technique that has been shown to improve the quality of machine translations, but is expensive, even if a sampling-based approximation is used. Besides requiring a large number of sampled sequences, it requires the pairwise calculation of a utility metric, which has quadratic complexity. In this paper, we propose to approximate pairwise metric scores with scores calculated against aggregated reference representations. This changes the complexity of utility estimation from $O(n^2)$ to $O(n)$, while empirically preserving most of the quality gains of MBR decoding. We release our source code at https://github.com/ZurichNLP/mbr

Figures (3)

• Figure 1: How accurately do MBR efficiency methods approximate standard MBR? In this validation experiment on newstest21, we gradually increase efficiency by using fewer references for pairwise utility estimation – either by subsampling the references (N-by-S; eikema-aziz-2022-sampling) or by aggregating their representations using partial aggregation (Section \ref{['sec:partial-aggregation']}). We report top-20 accuracy, which describes how often an efficiency method ranks the correct hypothesis (as selected by standard MBR) among the top 20 hypotheses. An efficiency method with a high top-20 accuracy could be used for pruning the number of hypotheses to 20 before standard MBR is applied.
• Figure 2: Top-1 accuracy of MBR efficiency methods on newstest21, analogous to Figure \ref{['fig:validation-top20-accuracy']}.
• Figure 3: Accuracy of MBR efficiency methods on newstest21 when using the Cometinho model rei-etal-2022-searching as utility metric.