Team ACK at SemEval-2025 Task 2: Beyond Word-for-Word Machine Translation for English-Korean Pairs

TL;DR

This study evaluates 13 models, including large language models and multilingual MT systems, on English-to-Korean translation for knowledge-intensive and entity-dense content using XC-Translate-derived data. It combines automatic metrics (BLEU, COMET, M-ETA) with human judgments to reveal gaps in current evaluation and translation capabilities, particularly for culturally nuanced entities. The findings show LLMs generally outperform traditional MT but struggle with entity-name translation and transliteration versus transcreation, prompting an error taxonomy that highlights key failure modes. The work underscores the need for more fine-grained, entity-aware evaluation and suggests directions for broader language-pair analysis and more culturally informed MT approaches.

Abstract

Translating knowledge-intensive and entity-rich text between English and Korean requires transcreation to preserve language-specific and cultural nuances beyond literal, phonetic or word-for-word conversion. We evaluate 13 models (LLMs and MT models) using automatic metrics and human assessment by bilingual annotators. Our findings show LLMs outperform traditional MT systems but struggle with entity translation requiring cultural adaptation. By constructing an error taxonomy, we identify incorrect responses and entity name errors as key issues, with performance varying by entity type and popularity level. This work exposes gaps in automatic evaluation metrics and hope to enable future work in completing culturally-nuanced machine translation.

Figures (4)

• Figure 1: Average BLEU, COMET, M-ETA scores by entity types.
• Figure 2: Frequency of errors per error taxonomy label.
• Figure 3: Average BLEU, COMET, M-ETA scores by popularity level.
• Figure 4: UI used for machine translation human annotation task.