Generating Difficult-to-Translate Texts

TL;DR

MT-breaker addresses the brittleness of machine translation benchmarks by leveraging an iterative, quality-estimation-guided text-breaking loop that uses an LLM to refine seeds based on MT outputs. It maintains source naturalness and diversity while increasing translation difficulty, with extensive experiments across languages, MT models, and LLMs, plus human evaluation confirming the findings. The approach reveals a trade-off between difficulty and diversity and demonstrates transfer of challenging cases across models and languages, though the generated data is model-specific and should be used for targeted diagnostics or cross-dataset comparisons under caution. Overall, MT-breaker provides a scalable framework for uncovering worst-case translation failures and informs dataset construction and model development beyond zero-shot or randomly sampled benchmarks.

Abstract

Machine translation benchmarks sourced from the real world are quickly obsoleted, due to most examples being easy for state-of-the-art translation models. This limits the benchmark's ability to distinguish which model is better or to reveal models' weaknesses. Current methods for creating difficult test cases, such as subsampling or from-scratch synthesis, either fall short of identifying difficult examples or suffer from a lack of diversity and naturalness. Inspired by the iterative process of human experts probing for model failures, we propose MT-breaker, a method where a large language model iteratively refines a source text to increase its translation difficulty. The LLM iteratively queries a target machine translation model to guide its generation of difficult examples. Our approach generates examples that are more challenging for the target MT model while preserving the diversity of natural texts. While the examples are tailored to a particular machine translation model during the generation, the difficulty also transfers to other models and languages.

Figures (5)

• Figure 1: High-level overview of our interactive difficulty-to-translate text generation pipeline. The process is repeated for a fixed number of iterations.
• Figure 1: Given a source text, the LLM receives the MT model's translation which informs how to next change the text. The $\mathrm{LLM}_\mathrm{step}$ can be instructed to not stray too far from the original meaning or to preserve naturalness. At the end, we pick the source text from $s$ that led to the worst translation.
• Figure 2: Development of difficulty with each iteration. Seeds require no iterations, Zeroshot and Zeroshot (history) require one LLM step, and the rest requires 10 steps. Right plot shows running hypotheses at a given step without final selection by quality estimation. This does not include Zeroshot (min) which is simply Zeroshot with selection. Values averaged across all models and languages. Shaded areas show 90% t-test confidence intervals averaged across all models and languages.
• Figure 3: Comparison of difficulty and diversity of difficult-to-translate text generation methods. Diversity is z-normalized across each diversity measure and averaged. Gray arrows signify most fair comparison.
• Figure 4: Results for difficulty for translation into English, compare to diagonal in \ref{['tab:02-transfer_language']}.