Simpson's Paradox and the Accuracy-Fluency Tradeoff in Translation

TL;DR

This work tackles the problem of how to reconcile translation accuracy and fluency. It introduces a formal framing with $\\text{accuracy}_M=\\log p(\\boldsymbol{x}|\\boldsymbol{y})$ and $\\text{fluency}_M=\\log p(\\boldsymbol{y})$, and demonstrates that Simpson's paradox yields a positive corpus-level association despite segment-level tradeoffs. The authors validate the paradox through theoretical formulation, Gaussian-simulated toy data, and empirical analyses of human and machine translations across CRITT, RLTC, MTMQM using a single NMT model to estimate the required probabilities, with an alternative verification using the M2M100 model in the appendix. The findings motivate segment-level evaluation and suggest balancing accuracy and fluency via noisy-channel objectives, with implications for both quality assessment protocols and MT system design.

Abstract

A good translation should be faithful to the source and should respect the norms of the target language. We address a theoretical puzzle about the relationship between these objectives. On one hand, intuition and some prior work suggest that accuracy and fluency should trade off against each other, and that capturing every detail of the source can only be achieved at the cost of fluency. On the other hand, quality assessment researchers often suggest that accuracy and fluency are highly correlated and difficult for human raters to distinguish (Callison-Burch et al., 2007). We show that the tension between these views is an instance of Simpson's paradox, and that accuracy and fluency are positively correlated at the level of the corpus but trade off at the level of individual source segments. We further suggest that the relationship between accuracy and fluency is best evaluated at the segment (or sentence) level, and that the trade off between these dimensions has implications both for assessing translation quality and developing improved MT systems.

Figures (7)

• Figure 1: Simpson's paradox. Each panel shows translations of three source segments indexed by color and marker shape. At the source segment level, accuracy and fluency (left) and $\text{accuracy}_M$ and $\text{fluency}_M$ (right, probabilities plotted on log scales) both show negative correlations $r_s$. At the corpus level, both pairs of dimensions show positive correlations $r_c$ (see panel labels). Significant correlations ($p<.05$) are marked with '*'. Source segments and translations are drawn from past WMT General Task submissions and data points have been jittered for clarity. The shaded areas show 95% confidence intervals based on 1000 bootstrapped samples. Full translations are included in Tables \ref{['tab:example1']} (orange dots), \ref{['tab:example3']} (green dots) and \ref{['tab:example2']} (blue dots) of the appendix.
• Figure 2: Tradeoffs between $p(\boldsymbol{x|y})$ and $p(\boldsymbol{y})$ in synthetic data.
• Figure 3: Tradeoffs between estimated $p(\boldsymbol{x|y})$ and $p(\boldsymbol{y})$ across source segments from three corpora. Paired-sample t-tests against randomly permuted $p(\boldsymbol{y})$ and $p(\boldsymbol{x|y})$ are shown at the top of each panel.
• Figure 4: Tradeoffs between human ratings of accuracy and fluency across segments from two corpora. Paired-sample t-tests against randomly permuted scores are shown at the top of each panel.
• Figure 5: $\text{accuracy}_M$ and $\text{fluency}_M$ predict human accuracy and fluency ratings for RLTC and WMT submissions to the general translation task in 2022 and 2023. zhen and ende refer to Chinese-English and English-German language pairs. All correlations reported are significant ($p<.001$).