Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agreement-Constrained Probabilistic Minimum Bayes Risk Decoding

Koki Natsumi, Hiroyuki Deguchi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

TL;DR

This work tackles the high computational cost of Minimum Bayes Risk decoding by leveraging Probabilistic MBR (PMBR), which partially observes utility scores and uses matrix completion, at the expense of translation quality. It introduces Agreement-Constrained PMBR (AC-PMBR), which ties the target metric to a knowledge-distilled metric through an agreement constraint to improve matrix completion under a fixed budget. Empirical results on WMT'23 En↔De show AC-PMBR yields up to 3x improvements in score-matrix MSE and gains in translation quality at comparable cost, especially under high reduction, demonstrating robustness and suggesting opportunities for multi-metric ensembles in decoding.

Abstract

Minimum Bayes risk (MBR) decoding generates high-quality translations by maximizing the expected utility of output candidates, but it evaluates all pairwise scores over the candidate set; hence, it takes quadratic time with respect to the number of candidates. To reduce the number of utility function calls, probabilistic MBR (PMBR) decoding partially evaluates quality scores using sampled pairs of candidates and completes the missing scores with a matrix completion algorithm. Nevertheless, it degrades the translation quality as the number of utility function calls is reduced. Therefore, to improve the trade-off between quality and cost, we propose agreement-constrained PMBR (AC-PMBR) decoding, which leverages a knowledge distilled model to guide the completion of the score matrix. Our AC-PMBR decoding improved approximation errors of matrix completion by up to 3 times and achieved higher translation quality compared with PMBR decoding at a comparable computational cost on the WMT'23 En$\leftrightarrow$De translation tasks.

Agreement-Constrained Probabilistic Minimum Bayes Risk Decoding

TL;DR

This work tackles the high computational cost of Minimum Bayes Risk decoding by leveraging Probabilistic MBR (PMBR), which partially observes utility scores and uses matrix completion, at the expense of translation quality. It introduces Agreement-Constrained PMBR (AC-PMBR), which ties the target metric to a knowledge-distilled metric through an agreement constraint to improve matrix completion under a fixed budget. Empirical results on WMT'23 En↔De show AC-PMBR yields up to 3x improvements in score-matrix MSE and gains in translation quality at comparable cost, especially under high reduction, demonstrating robustness and suggesting opportunities for multi-metric ensembles in decoding.

Abstract

Minimum Bayes risk (MBR) decoding generates high-quality translations by maximizing the expected utility of output candidates, but it evaluates all pairwise scores over the candidate set; hence, it takes quadratic time with respect to the number of candidates. To reduce the number of utility function calls, probabilistic MBR (PMBR) decoding partially evaluates quality scores using sampled pairs of candidates and completes the missing scores with a matrix completion algorithm. Nevertheless, it degrades the translation quality as the number of utility function calls is reduced. Therefore, to improve the trade-off between quality and cost, we propose agreement-constrained PMBR (AC-PMBR) decoding, which leverages a knowledge distilled model to guide the completion of the score matrix. Our AC-PMBR decoding improved approximation errors of matrix completion by up to 3 times and achieved higher translation quality compared with PMBR decoding at a comparable computational cost on the WMT'23 EnDe translation tasks.

Paper Structure

This paper contains 28 sections, 10 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. MBR decoding
  4. PMBR decoding
  5. Agreement-Constrained PMBR decoding
  6. Score matrix construction
  7. Agreement-constrained matrix completion
  8. Experimental Settings
  9. Evaluation
  10. Methods
  11. Utility function
  12. Hyperparameter
  13. Experimental Results and Discussions
  14. Translation quality
  15. Matrix completion accuracy
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Overview of our proposed Agreement-Constrained PMBR (AC-PMBR) decoding.
  • Figure 2: MSE of score matrices when varying computational costs in WMT'23 En$\rightarrow$De. #parameters refers to the number of parameters in the evaluation metric model, serving as an indicator of model scale, while the computational cost is described by Equation \ref{['eq:costPMBR']} and \ref{['eq:costACPMBR']}.
  • Figure 3: Agreement weight tuning in the low reduction rate setting on WMT'22 En$\rightarrow$De.
  • Figure 4: Agreement weight tuning in the high-reduction setting on WMT'22 En$\rightarrow$De.