DMDTEval: An Evaluation and Analysis of LLMs on Disambiguation in Multi-domain Translation

TL;DR

DMDTEval introduces a systematic benchmark to evaluate how well LLMs disambiguate lexical meaning in multi-domain translation (MDT). It builds an ambiguity-focused test set across 13 domains and four language pairs, designs a suite of disambiguation prompting templates (including zero-shot, chain-of-thought, few-shot, and reflection), and gauges performance with multiple open-source LLMs using translation quality metrics and a dedicated disambiguation accuracy measure. Key findings show domain-aware prompting, especially CoT and reflection strategies, can improve MDT translation, though effects vary by domain and metric, underscoring the need for specialized disambiguation metrics alongside traditional MT scores. The work provides an open dataset and actionable insights to advance domain-aware translation, with broader implications for evaluating and improving LLMs on lexical disambiguation in MDT scenarios.

Abstract

Currently, Large Language Models (LLMs) have achieved remarkable results in machine translation. However, their performance in multi-domain translation (MDT) is less satisfactory, the meanings of words can vary across different domains, highlighting the significant ambiguity inherent in MDT. Therefore, evaluating the disambiguation ability of LLMs in MDT, remains an open problem. To this end, we present an evaluation and analysis of LLMs on disambiguation in multi-domain translation (DMDTEval), our systematic evaluation framework consisting of three critical aspects: (1) we construct a translation test set with multi-domain ambiguous word annotation, (2) we curate a diverse set of disambiguation prompt strategies, and (3) we design precise disambiguation metrics, and study the efficacy of various prompt strategies on multiple state-of-the-art LLMs. We conduct comprehensive experiments across 4 language pairs and 13 domains, our extensive experiments reveal a number of crucial findings that we believe will pave the way and also facilitate further research in the critical area of improving the disambiguation of LLMs.

Figures (8)

• Figure 1: Two examples from the UM-Corpus English-Chinese test set. We prompt LLMs with domain label to disambiguate in Qwen-2.5-7B-Instruct. Red text represents for the ambiguity translation. Blue text represents for the correct translation (hereinafter the same).
• Figure 2: Ambiguous word test set construction annotation. This process consists of three steps.
• Figure 3: Statistics of ambiguous word in the test set.
• Figure 4: Design of Prompt Strategies. Light blue text represents for the specific information in each prompt strategy. Light green text represents for the specific information of disambiguation prompt strategies.
• Figure 5: The comparison of different LLMs on the English-to-Chinese translation task (T1–T4) in terms of average BLEU and COMET scores.