Benchmarking Machine Translation with Cultural Awareness

TL;DR

This work tackles the challenge of translating culture-specific items by creating CAMT, a CSI-annotated parallel corpus across six language pairs, and introducing two evaluation metrics—CSI-Match and PTA—for assessing cultural and pragmatic translation quality. It examines both traditional NMT and LLM-based MT, including manipulation via dictionary-based methods and prompting strategies that inject external cultural knowledge. The findings show that LLMs can effectively incorporate such external knowledge to improve pragmatic CSI translations, especially for CSIs without established translations, though performance varies by language resource. CAMT thus provides a scalable benchmark and actionable insights for advancing culturally-aware translation, with practical implications for cross-cultural information access and localization efforts.

Abstract

Translating culture-related content is vital for effective cross-cultural communication. However, many culture-specific items (CSIs) often lack viable translations across languages, making it challenging to collect high-quality, diverse parallel corpora with CSI annotations. This difficulty hinders the analysis of cultural awareness of machine translation (MT) systems, including traditional neural MT and the emerging MT paradigm using large language models (LLM). To address this gap, we introduce a novel parallel corpus, enriched with CSI annotations in 6 language pairs for investigating Culturally-Aware Machine Translation--CAMT. Furthermore, we design two evaluation metrics to assess CSI translations, focusing on their pragmatic translation quality. Our findings show the superior ability of LLMs over neural MTs in leveraging external cultural knowledge for translating CSIs, especially those lacking translations in the target culture.

Figures (10)

• Figure 1: Culture-specific item translation errors.
• Figure 2: Overview of CAMT construction pipeline.
• Figure 3: CSI-Match results on six language pairs.
• Figure 4: PTA results on English-Chinese translations.
• Figure 5: Category distribution on categories.