Large Language Models for Classical Chinese Poetry Translation: Benchmarking, Evaluating, and Improving

TL;DR

This work tackles the challenge of translating classical Chinese poetry by addressing adequacy, fluency, and elegance. It introduces the PoetMT benchmark to evaluate translations along these dimensions, alongside a GPT-4–based metric aligned with human judgments. The Retrieval-Augmented Translation RAT framework leverages a classical poetry knowledge base to produce more adequate, fluent, and elegant translations, outperforming multiple baselines across traditional and LLM-based metrics. The results highlight the importance of structured knowledge retrieval for poetry translation and provide a concrete path for improving LLM capabilities in culturally rich text translation.

Abstract

Different from the traditional translation tasks, classical Chinese poetry translation requires both adequacy and fluency in translating culturally and historically significant content and linguistic poetic elegance. Large language models (LLMs) with impressive multilingual capabilities may bring a ray of hope to achieve this extreme translation demand. This paper first introduces a suitable benchmark (PoetMT) where each Chinese poetry has a recognized elegant translation. Meanwhile, we propose a new metric based on GPT-4 to evaluate the extent to which current LLMs can meet these demands. Our empirical evaluation reveals that the existing LLMs fall short in the challenging task. Hence, we propose a Retrieval-Augmented Machine Translation (RAT) method which incorporates knowledge related to classical poetry for advancing the translation of Chinese Poetry in LLMs. Experimental results show that RAT consistently outperforms all comparison methods regarding wildly used BLEU, COMET, BLEURT, our proposed metric, and human evaluation.

Figures (10)

• Figure 1: An example block in the fluency and elegance in discourse-level poetry translation. The red parts indicate rhymes in both English and Chinese.
• Figure 2: An example block in the adequacy in sentence-level poetry translation.
• Figure 3: Examples of different evaluation metrics. Figure (a) represents the rhyme of the final words; Figure (b) shows that the two translated sentences have the same word count and couplet structure; Figure (c) indicates the accurate translation capturing the implied meaning of time passing.
• Figure 4: The proposed RAT framework. The "Historical Background," "Author Introduction," and "Modern Chinese Analysis" parts are at the discourse level, so the Selector needs to make selections based on the content.
• Figure 5: Experiment on the Impact of Different Knowledge of Classical Chinese Poetry on Translation. The dashed line indicates not using knowledge, but directly translating the result through ChatGPT.