GPT-4 vs. Human Translators: A Comprehensive Evaluation of Translation Quality Across Languages, Domains, and Expertise Levels

TL;DR

This work systematically evaluates GPT-4 against human translators across three language directions and three domains using a Multidimensional Quality Metrics (MQM) framework with expert annotators. It finds that GPT-4 matches junior translators in total errors but lags behind medium and senior translators, with notable imbalances across resource-rich versus resource-poor languages and domains. Qualitative analysis shows GPT-4 tends toward more literal translations, while humans sometimes over-interpret or imagine missing background information. The study highlights current limits of LLM-based translation and suggests opportunities for collaboration between GPT-4 and human translators to improve efficiency and quality in real-world workflows.

Abstract

This study comprehensively evaluates the translation quality of Large Language Models (LLMs), specifically GPT-4, against human translators of varying expertise levels across multiple language pairs and domains. Through carefully designed annotation rounds, we find that GPT-4 performs comparably to junior translators in terms of total errors made but lags behind medium and senior translators. We also observe the imbalanced performance across different languages and domains, with GPT-4's translation capability gradually weakening from resource-rich to resource-poor directions. In addition, we qualitatively study the translation given by GPT-4 and human translators, and find that GPT-4 translator suffers from literal translations, but human translators sometimes overthink the background information. To our knowledge, this study is the first to evaluate LLMs against human translators and analyze the systematic differences between their outputs, providing valuable insights into the current state of LLM-based translation and its potential limitations.

Figures (5)

• Figure 1: Upper: Error severity for each system. The gray line represents the standard deviation for each system across tasks. Bottom: Error category analysis for each system.
• Figure 2: Top 5 categories of errors made by each system.
• Figure 3: Error category results for each language. Each sub-figure is the average over two directions. We only include 'Major' errors here to highlight the most severe problems. Higher values indicate more errors and the number after each error type is the maximum number of that error.
• Figure 4: Error category results for different domains in Chinese-to-English. We only include 'Major' errors here to highlight the most severe problems. Higher values indicate more errors and the number in the bracket is the maximum number of that error.
• Figure 5: A screenshot of the Doccano annotation system we use.