Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Proverbs Run in Pairs: Evaluating Proverb Translation Capability of Large Language Model

Minghan Wang, Viet-Thanh Pham, Farhad Moghimifar, Thuy-Trang Vu

TL;DR

This work tackles proverb translation, a culturally nuanced MT challenge, by constructing two datasets (an English-centric standalone MAPS-based proverb set across five languages and a Proverb in Conversation (PiC) corpus mined from OpenSubtitles) and evaluating state-of-the-art NMT and diverse LLMs. It systematically investigates prompt design and conversational context, showing that LLMs generally outperform NMT, particularly when translating between languages from similar cultural regions, and that context-rich, dialogue-style prompts yield the strongest gains. The study also reveals that conventional automatic metrics (BLEU, CHRF++, COMET) struggle to capture proverb quality, and even LLM-based judgments have limitations, prompting calls for culturally aware evaluation methods. Overall, the results underscore the potential of LLMs for proverb translation while highlighting the need for larger datasets and better evaluation frameworks to reliably measure cultural nuance in MT.

Abstract

Despite achieving remarkable performance, machine translation (MT) research remains underexplored in terms of translating cultural elements in languages, such as idioms, proverbs, and colloquial expressions. This paper investigates the capability of state-of-the-art neural machine translation (NMT) and large language models (LLMs) in translating proverbs, which are deeply rooted in cultural contexts. We construct a translation dataset of standalone proverbs and proverbs in conversation for four language pairs. Our experiments show that the studied models can achieve good translation between languages with similar cultural backgrounds, and LLMs generally outperform NMT models in proverb translation. Furthermore, we find that current automatic evaluation metrics such as BLEU, CHRF++ and COMET are inadequate for reliably assessing the quality of proverb translation, highlighting the need for more culturally aware evaluation metrics.

Proverbs Run in Pairs: Evaluating Proverb Translation Capability of Large Language Model

TL;DR

This work tackles proverb translation, a culturally nuanced MT challenge, by constructing two datasets (an English-centric standalone MAPS-based proverb set across five languages and a Proverb in Conversation (PiC) corpus mined from OpenSubtitles) and evaluating state-of-the-art NMT and diverse LLMs. It systematically investigates prompt design and conversational context, showing that LLMs generally outperform NMT, particularly when translating between languages from similar cultural regions, and that context-rich, dialogue-style prompts yield the strongest gains. The study also reveals that conventional automatic metrics (BLEU, CHRF++, COMET) struggle to capture proverb quality, and even LLM-based judgments have limitations, prompting calls for culturally aware evaluation methods. Overall, the results underscore the potential of LLMs for proverb translation while highlighting the need for larger datasets and better evaluation frameworks to reliably measure cultural nuance in MT.

Abstract

Despite achieving remarkable performance, machine translation (MT) research remains underexplored in terms of translating cultural elements in languages, such as idioms, proverbs, and colloquial expressions. This paper investigates the capability of state-of-the-art neural machine translation (NMT) and large language models (LLMs) in translating proverbs, which are deeply rooted in cultural contexts. We construct a translation dataset of standalone proverbs and proverbs in conversation for four language pairs. Our experiments show that the studied models can achieve good translation between languages with similar cultural backgrounds, and LLMs generally outperform NMT models in proverb translation. Furthermore, we find that current automatic evaluation metrics such as BLEU, CHRF++ and COMET are inadequate for reliably assessing the quality of proverb translation, highlighting the need for more culturally aware evaluation metrics.
Paper Structure (38 sections, 1 equation, 7 figures, 9 tables)

This paper contains 38 sections, 1 equation, 7 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Proverb Translation Dataset
  3. Standalone Proverb Translation Dataset
  4. Proverb in Conversation Mining
  5. Initial Mining with Source-side Proverb
  6. Fine-grained Filtering
  7. LLM-based Proverb Usage Filtering
  8. Filtering with Quality Estimation
  9. Conversation Context Retrieval
  10. Human Evaluation
  11. Experimental Setup
  12. Models
  13. Prompts
  14. Evaluation
  15. Evaluation Metrics
  16. ...and 23 more sections

Figures (7)

  • Figure 1: Result on proverb translation of each translation direction.
  • Figure 2: We present the win rate of 5 pairs of models evaluated by the gpt-4o mini. Results are separated with From/To-En and Figurative/Literal, with corresponding COMET scores indicated near the model name.
  • Figure 3: Proverb Translation Annotation Instruction
  • Figure 4: Proverb Translation Annotation Interface
  • Figure 5: OpenSubtitle Translation Annotation Interface
  • ...and 2 more figures