Translation as a Scalable Proxy for Multilingual Evaluation

TL;DR

This work addresses the evaluation bottleneck in multilingual NLP by testing whether translation quality can serve as a scalable proxy for broader multilingual capabilities. Through a large-scale study of 14 LLMs, 9 benchmarks, and 3 translation datasets using 7 MT metrics, the authors demonstrate that translation quality correlates strongly with downstream multilingual performance (e.g., median $r$ values around $0.89$–$0.91$ for neural metrics). The findings support a two-stage evaluation workflow where translation-based screening rapidly screens thousands of language-model pairs, followed by targeted, task-specific benchmarks for cases with weaker or high-stakes requirements. They also delineate the proxy’s limits, noting stronger correlations for semantic understanding tasks than for specialized reasoning or domain-specific tasks, and call for careful, transparent use to avoid translation-centered biases while expanding evaluation to low-resource languages. Overall, translation quality emerges as a practical, cost-effective first-pass tool to broaden multilingual evaluation coverage while highlighting the need for task-specific follow-up and methodological safeguards.

Abstract

The rapid proliferation of LLMs has created a critical evaluation paradox: while LLMs claim multilingual proficiency, comprehensive non-machine-translated benchmarks exist for fewer than 30 languages, leaving >98% of the world's 7,000 languages in an empirical void. Traditional benchmark construction faces scaling challenges such as cost, scarcity of domain experts, and data contamination. We evaluate the validity of a simpler alternative: can translation quality alone indicate a model's broader multilingual capabilities? Through systematic evaluation of 14 models (1B-72B parameters) across 9 diverse benchmarks and 7 translation metrics, we find that translation performance is a good indicator of downstream task success (e.g., Phi-4, median Pearson r: MetricX = 0.89, xCOMET = 0.91, SSA-COMET = 0.87). These results suggest that the representational abilities supporting faithful translation overlap with those required for multilingual understanding. Translation quality, thus emerges as a strong, inexpensive first-pass proxy of multilingual performance, enabling a translation-first screening with targeted follow-up for specific tasks.

Figures (12)

• Figure 1: Pearson correlation matrix for the Phi-4 (14B) model, showing correlations between MT quality metrics and multilingual benchmark performance on FLORES-200. Color intensity indicates the strength of correlation, with darker red denoting stronger positive correlations.
• Figure 2: Aggregated Pearson correlations between MT metrics and multilingual benchmarks across 14 models ranging from 1B to 72B parameters on the Flores-200 dataset. Left: Correlations with Multiple Choice (MC) benchmarks. Right: Correlations with Generative benchmarks. Shaded areas indicate 95% confidence intervals. Red lines represent model-based neural MT metrics, while blue lines represent parametric metrics.
• Figure 3: Correlation between translation quality (SSA-COMET) and reading comprehension (Belebele) for Phi-4 across 115 languages. The high coefficients ($r=0.936, \rho=0.943$) demonstrate a strong linear alignment between translation and NLU tasks.
• Figure 4: Correlation between translation quality (xCOMET) and commonsense reasoning (HellaSwag) for Phi-4 across 30 languages. The alignment ($r=0.972, \rho=0.977$) suggests that the semantic representations required for translation and commonsense reasoning are highly congruent.
• Figure 5: WMT Correlation