DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translation

TL;DR

DITING introduces a six-dimension evaluation framework to assess narrative and cultural fidelity in CN–EN web-novel translation, addressing idioms, lexical ambiguity, terminology localization, tense coherence, zero pronouns, and safety. It pairs with AgentEval, a reasoning-driven multi-agent evaluation that simulates expert deliberation, and MetricAlign, a meta-evaluation dataset to benchmark automatic metrics. Across fourteen translation models, the study finds Chinese-trained LLMs outperform larger foreign models, with DeepSeek-V3 delivering the most faithful and stylistically coherent translations. The work provides public datasets and a new paradigm for evaluating LLM-based web-novel translation, highlighting remaining gaps in cultural safety and discourse-level coherence.

Abstract

Large language models (LLMs) have substantially advanced machine translation (MT), yet their effectiveness in translating web novels remains unclear. Existing benchmarks rely on surface-level metrics that fail to capture the distinctive traits of this genre. To address these gaps, we introduce DITING, the first comprehensive evaluation framework for web novel translation, assessing narrative and cultural fidelity across six dimensions: idiom translation, lexical ambiguity, terminology localization, tense consistency, zero-pronoun resolution, and cultural safety, supported by over 18K expert-annotated Chinese-English sentence pairs. We further propose AgentEval, a reasoning-driven multi-agent evaluation framework that simulates expert deliberation to assess translation quality beyond lexical overlap, achieving the highest correlation with human judgments among seven tested automatic metrics. To enable metric comparison, we develop MetricAlign, a meta-evaluation dataset of 300 sentence pairs annotated with error labels and scalar quality scores. Comprehensive evaluation of fourteen open, closed, and commercial models reveals that Chinese-trained LLMs surpass larger foreign counterparts, and that DeepSeek-V3 delivers the most faithful and stylistically coherent translations. Our work establishes a new paradigm for exploring LLM-based web novel translation and provides public resources to advance future research.

Figures (3)

• Figure 1: Examples of ground truth and low-quality translations across six dimensions, showing that even translations with high BLEU scores can contain errors causing reader confusion and misinterpretation.
• Figure 2: Overview of our work.
• Figure 3: The Label Studio interface of the DITING annotation process.