Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluation

TL;DR

This work challenges MAP-based decoding in reference-free error span detection and introduces Minimum Bayes Risk (MBR) decoding with three utilities, notably SoftF1, to better align MT ESD outputs with human annotations. By generating diverse candidate hypotheses and optimizing expected utility, MBR-SoftF1 achieves strong span-level improvements on the WMT24 Metrics Shared Task and often matches or surpasses MAP at system and sentence levels. A key contribution is the SoftF1 utility, which enables soft evaluation and robust ranking even with empty references, and the demonstration that MBR distillation can deliver MBR-like performance with greedy decoding, offering practical latency benefits. The findings suggest that MBR decoding, especially with SoftF1 and adequate candidate sets, provides a significant advancement for diagnostic MT evaluation and enables more faithful ESD analyses and downstream post-editing workflows.

Abstract

Error Span Detection (ESD) extends automatic machine translation (MT) evaluation by localizing translation errors and labeling their severity. Current generative ESD methods typically use Maximum a Posteriori (MAP) decoding, assuming that the model-estimated probabilities are perfectly correlated with similarity to the human annotation, but we often observe higher likelihood assigned to an incorrect annotation than to the human one. We instead apply Minimum Bayes Risk (MBR) decoding to generative ESD. We use a sentence- or span-level similarity function for MBR decoding, which selects candidate hypotheses based on their approximate similarity to the human annotation. Experimental results on the WMT24 Metrics Shared Task show that MBR decoding significantly improves span-level performance and generally matches or outperforms MAP at the system and sentence levels. To reduce the computational cost of MBR decoding, we further distill its decisions into a model decoded via greedy search, removing the inference-time latency bottleneck.

Figures (2)

• Figure 1: Performance of decoding methods with Llama-3.3-70B-Inst on the WMT24 Metrics Shared Task. MBR indicates our MBR-SoftF1.
• Figure 2: An overview of our MBR decoding for generative ESD models. Error spans are highlighted in red.