Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Models

Tobias Domhan, Dawei Zhu

TL;DR

The paper investigates how input length affects translation evaluation with large language models and discovers a pronounced length bias: longer inputs yield fewer MQM error spans and lower system ranking. It proposes three length-invariant strategies—Focus Sentence Prompting (FSP), granularity-matching prompts (GMICL-5/GMFT), and error-span explanations—alongside Direct Assessment (DA). Empirical results on WMT'24 data across three language directions show that FSP most effectively stabilizes error detection and improves ranking, with GMFT offering additional gains; DA alone is insufficient but can benefit from integration with FSP. The work provides practical guidelines for deploying LLMs in long-form translation evaluation, balancing accuracy, cost, and data needs, while acknowledging limitations such as direction coverage and scope of document-level phenomena.

Abstract

Accurately evaluating machine-translated text remains a long-standing challenge, particularly for long documents. Recent work has shown that large language models (LLMs) can serve as reliable and interpretable sentence-level translation evaluators via MQM error span annotations. With modern LLMs supporting larger context windows, a natural question arises: can we feed entire document translations into an LLM for quality assessment? Ideally, evaluation should be invariant to text length, producing consistent error spans regardless of input granularity. However, our analysis shows that text length significantly impacts evaluation: longer texts lead to fewer error spans and reduced system ranking accuracy. To address this limitation, we evaluate several strategies, including granularity-aligned prompting, Focus Sentence Prompting (FSP), and a fine-tuning approach to better align LLMs with the evaluation task. The latter two methods largely mitigate this length bias, making LLMs more reliable for long-form translation evaluation.

Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Models

TL;DR

The paper investigates how input length affects translation evaluation with large language models and discovers a pronounced length bias: longer inputs yield fewer MQM error spans and lower system ranking. It proposes three length-invariant strategies—Focus Sentence Prompting (FSP), granularity-matching prompts (GMICL-5/GMFT), and error-span explanations—alongside Direct Assessment (DA). Empirical results on WMT'24 data across three language directions show that FSP most effectively stabilizes error detection and improves ranking, with GMFT offering additional gains; DA alone is insufficient but can benefit from integration with FSP. The work provides practical guidelines for deploying LLMs in long-form translation evaluation, balancing accuracy, cost, and data needs, while acknowledging limitations such as direction coverage and scope of document-level phenomena.

Abstract

Accurately evaluating machine-translated text remains a long-standing challenge, particularly for long documents. Recent work has shown that large language models (LLMs) can serve as reliable and interpretable sentence-level translation evaluators via MQM error span annotations. With modern LLMs supporting larger context windows, a natural question arises: can we feed entire document translations into an LLM for quality assessment? Ideally, evaluation should be invariant to text length, producing consistent error spans regardless of input granularity. However, our analysis shows that text length significantly impacts evaluation: longer texts lead to fewer error spans and reduced system ranking accuracy. To address this limitation, we evaluate several strategies, including granularity-aligned prompting, Focus Sentence Prompting (FSP), and a fine-tuning approach to better align LLMs with the evaluation task. The latter two methods largely mitigate this length bias, making LLMs more reliable for long-form translation evaluation.
Paper Structure (30 sections, 14 figures, 3 tables)

This paper contains 30 sections, 14 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Length-invariant translation evaluation
  3. Focus sentence prompting (FSP)
  4. Granularity Matching
  5. Error Span Explanations
  6. Direct Assessment (DA)
  7. Experiments
  8. Setup
  9. Addressing evaluation length dependence
  10. FSP inference efficiency
  11. Related work
  12. Conclusion
  13. Limited Translation Directions
  14. Evaluation Scope
  15. Sentence Segmentation Requirement
  16. ...and 15 more sections

Figures (14)

  • Figure 1: The number of predicted errors (left) and the average accuracy (right) for different input text granularities: segment-level, doc-level, and 5 doc-level.
  • Figure 2: Comparison of chat response lengths (chat) compared to Claude 3.5 Sonnet's MQM response length (seg, doc, 5doc) on WMT'24 EN-DE metrics task data in GPT-4o tokens. The expected length is based on the concatenation of the segment level responses.
  • Figure 3: FSP on a three-sentence document.
  • Figure 4: System ranking accuracy and number of error spans per document across methods and input granularities, averaged across translation directions. See Appendix \ref{['sec:translation_direction_specific_results']}, \ref{['sec:precision_recall_f1']} for direction-specific results and character F1 scores.
  • Figure 5: Impact of predicting a quality score via DA compared to computing a weighted MQM error score for GEMBA and FSP prompting.
  • ...and 9 more figures