Quality-Aware Translation Models: Efficient Generation and Quality Estimation in a Single Model

TL;DR

This work tackles the misalignment between neural translation model probabilities and human judgments by embedding quality estimation directly into the NMT system. It introduces two quality-aware strategies—Quality-Aware Prompting and Quality-Aware Prediction—that teach the model to generate and/or judge translation quality using discretized quality tokens, eliminating the need for external QE models at decoding time. Empirically, these methods deliver quality improvements comparable to or better than QE reranking while drastically reducing decoding cost, particularly when used with Minimum Bayes Risk decoding where candidate lists can be dramatically pruned. The approach not only boosts translation quality but also enables single-pass decoding with significant speedups, offering practical impact for high-resource languages and potential extensions to LLMs and multilingual/low-resource settings.

Abstract

Maximum-a-posteriori (MAP) decoding is the most widely used decoding strategy for neural machine translation (NMT) models. The underlying assumption is that model probability correlates well with human judgment, with better translations getting assigned a higher score by the model. However, research has shown that this assumption does not always hold, and generation quality can be improved by decoding to optimize a utility function backed by a metric or quality-estimation signal, as is done by Minimum Bayes Risk (MBR) or quality-aware decoding. The main disadvantage of these approaches is that they require an additional model to calculate the utility function during decoding, significantly increasing the computational cost. In this paper, we propose to make the NMT models themselves quality-aware by training them to estimate the quality of their own output. Using this approach for MBR decoding we can drastically reduce the size of the candidate list, resulting in a speed-up of two-orders of magnitude. When applying our method to MAP decoding we obtain quality gains similar or even superior to quality reranking approaches, but with the efficiency of single pass decoding.

Figures (6)

• Figure 1: Alignment between predicted quality scores from the QA Prediction model and actual MetricX-QE scores of translations in the en $\rightarrow$ de test dataset. The boxplots show the distribution of actual scores across all samples assigned to each bin. The median ground truth quality score increases steadily in line with the predicted bins.
• Figure 2: Translation quality dependent on the QA label used for prompting. Higher values in the label prompt the system to generate better translations.
• Figure 3: Performance of quality-aware approaches (QA Prediction and QA Prompting) compared to baseline MBR decoding across various candidate list sizes. MBR decoding with quality-aware models consistently outperforms baseline MBR decoding across candidate list sizes. The quality-aware approaches can achieve the same level of performance as baseline approaches while reducing the required utility function computations by up to two orders of magnitude.
• Figure 4: Performance of quality-aware approaches (Quality-Aware Prediction and Quality-Aware Prompting) compared to baseline MBR decoding across various candidate list sizes for $en \rightarrow ja$. MBR decoding with quality-aware models consistently outperforms baseline MBR decoding across candidate list sizes. The quality-aware approaches can achieve the same level of performance as baseline approaches while reducing the required utility function computations by one to two orders of magnitude.
• Figure 5: Sensitivity concerning performance of the Quality-Aware Prediction approach w.r.t. the number of quality score bins. Increasing the number of quality score bins yields generally improvements on our quality metrics, specifically in the range of 2 to 5 bins.