LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answering

TL;DR

This paper tackles the challenge of evaluating literary translations, where traditional metrics fail to capture aesthetic and cultural nuances. It introduces TransPro, a reference-free, QA-based metric that embeds professional translator input through weights and carefully designed questions, paired with an auxiliary finetuning of XCOMET-xl on literary data. Across LitEval-Corpus, LiteraryTran, and PAR3-annotated datasets, TransPro achieves notable gains in correlation (up to 0.07) and adequacy (over 15 points) compared with state-of-the-art metrics, and demonstrates reasonable performance with open-source base models, broadening accessibility. While TransPro approaches the performance of trained linguistics students on adequacy, it still trails expert translators, underscoring both the promise and the need for further domain-specific refinement and dataset expansion to reach professional-level evaluation in literary translation.

Abstract

The impact of Large Language Models (LLMs) has extended into literary domains. However, existing evaluation metrics for literature prioritize mechanical accuracy over artistic expression and tend to overrate machine translation as being superior to human translation from experienced professionals. In the long run, this bias could result in an irreversible decline in translation quality and cultural authenticity. In response to the urgent need for a specialized literary evaluation metric, we introduce LITRANSPROQA, a novel, reference-free, LLM-based question-answering framework designed for literary translation evaluation. LITRANSPROQA integrates humans in the loop to incorporate insights from professional literary translators and researchers, focusing on critical elements in literary quality assessment such as literary devices, cultural understanding, and authorial voice. Our extensive evaluation shows that while literary-finetuned XCOMET-XL yields marginal gains, LITRANSPROQA substantially outperforms current metrics, achieving up to 0.07 gain in correlation and surpassing the best state-of-the-art metrics by over 15 points in adequacy assessments. Incorporating professional translator insights as weights further improves performance, highlighting the value of translator inputs. Notably, LITRANSPROQA reaches an adequacy performance comparable to trained linguistic student evaluators, though it still falls behind experienced professional translators. LITRANSPROQA shows broad applicability to open-source models like LLaMA3.3-70b and Qwen2.5-32b, indicating its potential as an accessible and training-free tool for evaluating literary translations that require local processing due to copyright or ethical considerations.

Figures (5)

• Figure 1: Overview of TransPro and its performance compared to finetuned and LLM-based SOTA in correlation and adequacy (the ability to rate high-quality human translation better than MT). Human MQM (dashed lines) represents the adequacy level of trained students using MQM. Human level refers to trained student evaluators, who still lag behind experienced professional translators by a large margin according to Zhang et al. (2024).
• Figure 2: Screenshot of the instruction page for the survey.
• Figure 3: Pairwise inter-annotator agreement distribution for question importance ratings among the seven translators.
• Figure 4: Comparison of weighted vs. unweighted TransPro scores: average results across all 3 test datasets.
• Figure 5: Screenshot of survey page.