When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages

TL;DR

This work tackles reference-less segment-level quality estimation (DA scores on a scale of $0-100$) for machine translation in low-resource languages by evaluating large language models under zero-shot, few-shot, and instruction-finetuning regimes using an annotation-guidelines prompt. The AG-prompt improves zero-shot performance, while instruction fine-tuning narrows the gap with encoder-based QE models, though the latter remain stronger across most language pairs. A tokenization analysis reveals cross-lingual semantic matching is hindered by how LLMs tokenize morphologically rich languages, and error analysis highlights terminology, entities, and syntax as persistent challenges. The authors release QE data and trained models to enable further research, underscoring the need for better cross-lingual pre-training and tokenization for low-resource QE tasks and suggesting avenues like regression-head adapters for improved performance.

Abstract

This paper investigates the reference-less evaluation of machine translation for low-resource language pairs, known as quality estimation (QE). Segment-level QE is a challenging cross-lingual language understanding task that provides a quality score (0-100) to the translated output. We comprehensively evaluate large language models (LLMs) in zero/few-shot scenarios and perform instruction fine-tuning using a novel prompt based on annotation guidelines. Our results indicate that prompt-based approaches are outperformed by the encoder-based fine-tuned QE models. Our error analysis reveals tokenization issues, along with errors due to transliteration and named entities, and argues for refinement in LLM pre-training for cross-lingual tasks. We release the data, and models trained publicly for further research.

Figures (12)

• Figure 1: The proposed AG prompt which augments scoring instructions within the context.
• Figure 2: Best fine-tuned performance (Spearman) for LLMs vs. TransQuest-InfoXLM vs. COMET
• Figure 3: Error types and their percent contribution.
• Figure 4: The graphs compare the original word counts with the model-generated token counts for selected inputs, as described in Section \ref{['sec:tokenization']}. This comparison includes both low-resource language pairs (En-Ta, Et-En) and a high-resource language pair (En-De). A detailed image covering all language pairs is provided in the Appendix \ref{['app:token']}.
• Figure 5: Our proposed AG prompt for in-context learning.