M-Prometheus: A Suite of Open Multilingual LLM Judges

TL;DR

M-Prometheus addresses the lack of robust multilingual LLM judges by delivering an open-weight suite that provides both direct assessments and pairwise feedback across 20+ languages. The authors combine synthesized multilingual feedback with MT-evaluation data under Prometheus-2-inspired finetuning of Qwen2.5-Instruct, achieving state-of-the-art results on multilingual benchmarks and literary MT while preserving English performance. Extensive ablations identify backbone choice, synthetic multilingual data, and dataset composition as key drivers of performance, with translated data being less effective than synthetic multilingual data. The work also demonstrates practical utility through quality-aware decoding to improve multilingual outputs at inference and releases data and code to spur further research in multilingual evaluation.

Abstract

The use of language models for automatically evaluating long-form text (LLM-as-a-judge) is becoming increasingly common, yet most LLM judges are optimized exclusively for English, with strategies for enhancing their multilingual evaluation capabilities remaining largely unexplored in the current literature. This has created a disparity in the quality of automatic evaluation methods for non-English languages, ultimately hindering the development of models with better multilingual capabilities. To bridge this gap, we introduce M-Prometheus, a suite of open-weight LLM judges ranging from 3B to 14B parameters that can provide both direct assessment and pairwise comparison feedback on multilingual outputs. M-Prometheus models outperform state-of-the-art open LLM judges on multilingual reward benchmarks spanning more than 20 languages, as well as on literary machine translation (MT) evaluation covering 4 language pairs. Furthermore, M-Prometheus models can be leveraged at decoding time to significantly improve generated outputs across all 3 tested languages, showcasing their utility for the development of better multilingual models. Lastly, through extensive ablations, we identify the key factors for obtaining an effective multilingual judge, including backbone model selection and training on synthetic multilingual feedback data instead of translated data. We release our models, training dataset, and code.

Figures (13)

• Figure 1: M-Prometheus is a suite of open-weight multilingual LLM judges capable of providing reference-based and reference-free direct assessment and pairwise feedback.
• Figure 2: Data distribution (in number of instances) of the M-Feedback Collection (DA data) and M-Preference Collection (PWC data) datasets. These datasets form the training data of M-Prometheus.
• Figure 3: Performance of the M-Prometheus and Qwen2.5-Instruct 3B, 7B, and 14B models on general-purpose multilingual benchmarks, broken down by category. Tables with more detailed results are in Appendix \ref{['sec:apx-results-by-cat']}.
• Figure 4: Multilingual example of M-Feedback Collection part 1.
• Figure 5: Multilingual example of M-Feedback Collection part 2.