Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finding Replicable Human Evaluations via Stable Ranking Probability

Parker Riley, Daniel Deutsch, George Foster, Viresh Ratnakar, Ali Dabirmoghaddam, Markus Freitag

TL;DR

This paper tackles the challenge of obtaining stable, replicable human evaluations for ranking NLG systems by using MT with MQM as a case study. It introduces Stable Ranking Probability (SRP) as a meta-evaluation metric and combines a formal simulation framework with empirical MQM data from two language pairs to derive practical guidelines. Key contributions include concrete recommendations on item grouping, workload balance, score normalization, and ratings budgeting, plus the release of nearly 140,000 MQM segment ratings. The work advances reliable evaluation design for NLG and supports robust decision-making in model development and deployment.

Abstract

Reliable human evaluation is critical to the development of successful natural language generation models, but achieving it is notoriously difficult. Stability is a crucial requirement when ranking systems by quality: consistent ranking of systems across repeated evaluations is not just desirable, but essential. Without it, there is no reliable foundation for hill-climbing or product launch decisions. In this paper, we use machine translation and its state-of-the-art human evaluation framework, MQM, as a case study to understand how to set up reliable human evaluations that yield stable conclusions. We investigate the optimal configurations for item allocation to raters, number of ratings per item, and score normalization. Our study on two language pairs provides concrete recommendations for designing replicable human evaluation studies. We also collect and release the largest publicly available dataset of multi-segment translations rated by multiple professional translators, consisting of nearly 140,000 segment annotations across two language pairs.

Finding Replicable Human Evaluations via Stable Ranking Probability

TL;DR

This paper tackles the challenge of obtaining stable, replicable human evaluations for ranking NLG systems by using MT with MQM as a case study. It introduces Stable Ranking Probability (SRP) as a meta-evaluation metric and combines a formal simulation framework with empirical MQM data from two language pairs to derive practical guidelines. Key contributions include concrete recommendations on item grouping, workload balance, score normalization, and ratings budgeting, plus the release of nearly 140,000 MQM segment ratings. The work advances reliable evaluation design for NLG and supports robust decision-making in model development and deployment.

Abstract

Reliable human evaluation is critical to the development of successful natural language generation models, but achieving it is notoriously difficult. Stability is a crucial requirement when ranking systems by quality: consistent ranking of systems across repeated evaluations is not just desirable, but essential. Without it, there is no reliable foundation for hill-climbing or product launch decisions. In this paper, we use machine translation and its state-of-the-art human evaluation framework, MQM, as a case study to understand how to set up reliable human evaluations that yield stable conclusions. We investigate the optimal configurations for item allocation to raters, number of ratings per item, and score normalization. Our study on two language pairs provides concrete recommendations for designing replicable human evaluation studies. We also collect and release the largest publicly available dataset of multi-segment translations rated by multiple professional translators, consisting of nearly 140,000 segment annotations across two language pairs.
Paper Structure (27 sections, 1 equation, 8 figures, 4 tables)

This paper contains 27 sections, 1 equation, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Terminology
  3. Research Questions and Findings
  4. Data
  5. Background on MQM
  6. Dataset Descriptions
  7. Rater Behavior and Agreement
  8. Quantifying Replicability
  9. Simulation Analysis
  10. Evaluation Methodology Features
  11. How should rateable items be grouped?
  12. How should workload be distributed?
  13. How should scores be normalized?
  14. How many items should be annotated?
  15. How many ratings per item are needed?
  16. ...and 12 more sections

Figures (8)

  • Figure 1: Distribution of segment-level MQM scores for two English-Chinese raters who rated the same items, highlighting differences in rater behavior.
  • Figure 2: Overview of the simulated rater assignment procedure, focusing on two features. System translations are collected for the shuffled documents, and then optionally shuffled based on the Item Grouping feature. "By System" means that items are only shuffled among the positions from the same system. Items are then assigned to shuffled raters, with the (im)balance of the distribution controlled by the Load Balancing feature.
  • Figure 3: Evaluation of the Item Grouping feature.
  • Figure 4: Evaluation of the Load-Balancing feature. Our datasets show the opposite relationship between imbalance and stability.
  • Figure 5: Evaluation of the Normalization feature in our recommended setting.
  • ...and 3 more figures