Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control

Arle Lommel, Serge Gladkoff, Alan Melby, Sue Ellen Wright, Ingemar Strandvik, Katerina Gasova, Angelika Vaasa, Andy Benzo, Romina Marazzato Sparano, Monica Foresi, Johani Innis, Lifeng Han, Goran Nenadic

TL;DR

This paper addresses translation quality measurement across diverse sample sizes by extending the Multidimensional Quality Metrics (MQM) framework. It introduces MQM 2.0, combining a hierarchical Error Typology with three scoring families (Linear Raw, Linear with Calibration, Non-Linear with Calibration) and a multi-range approach that covers small to very large samples, guided by Statistical Quality Control for tiny samples. Central contributions include the MQM Core/Full typologies, calibrated scoring, and a non-linear scoring model that aligns scores with human perception and scales across sample sizes, plus practical guidance for setting up MQM evaluations and scorecards. The work has practical impact for industry benchmarking, standardization across content types, and aligning human-centric evaluation with automatic metrics and GenAI considerations.

Abstract

The year 2024 marks the 10th anniversary of the Multidimensional Quality Metrics (MQM) framework for analytic translation quality evaluation. The MQM error typology has been widely used by practitioners in the translation and localization industry and has served as the basis for many derivative projects. The annual Conference on Machine Translation (WMT) shared tasks on both human and automatic translation quality evaluations used the MQM error typology. The metric stands on two pillars: error typology and the scoring model. The scoring model calculates the quality score from annotation data, detailing how to convert error type and severity counts into numeric scores to determine if the content meets specifications. Previously, only the raw scoring model had been published. This April, the MQM Council published the Linear Calibrated Scoring Model, officially presented herein, along with the Non-Linear Scoring Model, which had not been published before. This paper details the latest MQM developments and presents a universal approach to translation quality measurement across three sample size ranges. It also explains why Statistical Quality Control should be used for very small sample sizes, starting from a single sentence.

The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control

TL;DR

This paper addresses translation quality measurement across diverse sample sizes by extending the Multidimensional Quality Metrics (MQM) framework. It introduces MQM 2.0, combining a hierarchical Error Typology with three scoring families (Linear Raw, Linear with Calibration, Non-Linear with Calibration) and a multi-range approach that covers small to very large samples, guided by Statistical Quality Control for tiny samples. Central contributions include the MQM Core/Full typologies, calibrated scoring, and a non-linear scoring model that aligns scores with human perception and scales across sample sizes, plus practical guidance for setting up MQM evaluations and scorecards. The work has practical impact for industry benchmarking, standardization across content types, and aligning human-centric evaluation with automatic metrics and GenAI considerations.

Abstract

The year 2024 marks the 10th anniversary of the Multidimensional Quality Metrics (MQM) framework for analytic translation quality evaluation. The MQM error typology has been widely used by practitioners in the translation and localization industry and has served as the basis for many derivative projects. The annual Conference on Machine Translation (WMT) shared tasks on both human and automatic translation quality evaluations used the MQM error typology. The metric stands on two pillars: error typology and the scoring model. The scoring model calculates the quality score from annotation data, detailing how to convert error type and severity counts into numeric scores to determine if the content meets specifications. Previously, only the raw scoring model had been published. This April, the MQM Council published the Linear Calibrated Scoring Model, officially presented herein, along with the Non-Linear Scoring Model, which had not been published before. This paper details the latest MQM developments and presents a universal approach to translation quality measurement across three sample size ranges. It also explains why Statistical Quality Control should be used for very small sample sizes, starting from a single sentence.
Paper Structure (42 sections, 11 figures)

This paper contains 42 sections, 11 figures.

Table of Contents

  1. Introduction and Background
  2. On Sampling
  3. Low IRR is not a bug, but a feature
  4. Three sample-size ranges -- three very different methods
  5. MQM 2.0: The State of the Art of Analytical Quality Evaluation
  6. Error Typology
  7. Scoring Model
  8. Setting up an MQM evaluation system
  9. Projecting PI and the Formulas
  10. On Non-Linearity
  11. Non-Linear error tolerance – what it may look like?
  12. Benefits of non-linear scoring: faithfulness to human perception and scalable to a wide range of sample sizes
  13. Conclusion and Future Work
  14. Parameters of the Evaluation Quality Metric
  15. Calculation Values
  16. ...and 27 more sections

Figures (11)

  • Figure 1: MQM2.0 Evaluation Scorecard: Quality Measure and Tools (red squares with straight corners), and Quality Scores (orange squares with curved corners).
  • Figure 2: Projecting the small Passing Interval window in the Raw-Score scale to the scale of the Calibrated Score, where the Passing Threshold is chosen arbitrarily by the customer based on the relevant values that apply to a specific context.
  • Figure 3: Formulas for raw score calculation.
  • Figure 4: Formulas for calibrated score calculation.
  • Figure 5: Extended calibration questionnaire.
  • ...and 6 more figures