From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set

TL;DR

The paper introduces Specialist, a method to specialize LLM-based Autoraters to a fixed test set by constructing per-example ICL demonstrations from historical ratings. This specialist AutoMQM achieves state-of-the-art fine-grained MT evaluation on WMT'23 and WMT'24, outperforming XCOMET by substantial margins and demonstrating robustness across LLM backbones and evaluation tasks. The approach leverages pseudo-SxS rating collection, demonstrates scaling behavior with the number of ICL demonstrations, and provides interpretability through span-based MQM errors, with implications for broader NLG evaluation and potential extensions to other tasks.

Abstract

As LLMs continue to become more powerful and versatile, human evaluation has quickly become intractable at scale and reliance on automatic metrics has become the norm. Recently, it has been shown that LLMs are themselves state-of-the-art evaluators for many tasks. These Autoraters are typically designed so that they generalize to new systems and test sets. In practice, however, evaluation is performed on a small set of fixed, canonical test sets, which are carefully curated to measure certain capabilities of interest and are not changed frequently. In this work, we design a method which specializes a prompted Autorater to a given test set, by leveraging historical ratings on the test set to construct in-context learning (ICL) examples. We evaluate our Specialist method on the task of fine-grained machine translation evaluation, and show that it dramatically outperforms the state-of-the-art XCOMET metric by 54% and 119% on the WMT'23 and WMT'24 test sets, respectively. We perform extensive analyses to understand the representations learned by our Specialist metrics, and how variability in rater behavior affects their performance. We also verify the generalizability and robustness of our Specialist method for designing automatic metrics across different numbers of ICL examples, LLM backbones, systems to evaluate, and evaluation tasks.

Figures (6)

• Figure 1: Illustration of the Specialist method, compared against the Fixed, differentsource baseline, for prompting an LLM-based Autorater. Both methods (i) construct a unique set of demonstrations (i.e., ICL examples) for every test set example, consisting of historical ratings from different system outputs for some fixed source, and (ii) provide demonstrations from the same rater as the test rating ground truth. The difference between these methods is that the Specialist ICL examples consist of ratings of outputs from the same source as the test example.
• Figure 2: Specialized AutoMQM performance per translation system. The Champion (row 2a) and Shuffled baseline (row 1c) models from Table \ref{['tab:main_table_wmt23']} are compared. The average MQM score of each system, along with its name, is shown in the x-axis. The average character-level F1 of the AutoMQM model when evaluated on this system only is shown on the y-axis.
• Figure 3: Specialist AutoMQM performance ("Champion + Filter" setting; Table \ref{['tab:main_table_wmt23']}, row 2b) as a function of number of ICL examples used. For comparison, XCOMET performance, as well as ICL example scaling for the "Shuffled sources" baseline (Table \ref{['tab:main_table_wmt23']}, row 1c), are also shown.
• Figure 4: Cross-rater performance of AutoMQM and human annotators, computed using the extension to the (round 1) WMT'23 zh$\rightarrow$en test set, whereby 10% of the test set (18 source segments $\times$ 15 systems = 270 examples) was rated by all 8 raters. In Figure (a), Specialist AutoMQM is prompted with (same-source) ICL examples from the icl_rater_id rater (vertical axis), and evaluated using the ratings from the test_rater_id rater (horizontal axis). This figure shows the entire matrix of character-level F1 scores for every (icl_rater_id, test_rater_id) pair. In Figure (b), the matrix of character-level F1 scores between all pairs of human annotators is shown.
• Figure 5: AutoMQM prompt, with placeholders for {source_language}, {source} (for both ICL examples and the test example), {target_language}, {translation} (again, for both ICL examples and the test example), and {errors in JSON format} (for ICL examples only).