LangMark: A Multilingual Dataset for Automatic Post-Editing

TL;DR

LangMark provides a large multilingual, human-annotated dataset for automatic post-editing (APE) on NMT outputs, addressing the scarcity of multilingual training resources. The dataset covers English-to-seven languages with 206,983 triplets, enabling rigorous benchmarking. The authors show that large language models with few-shot prompting can improve MT outputs and even surpass proprietary MT systems in some cases, while also analyzing the task of deciding when edits are needed. This resource and analysis highlight the need for robust evaluation metrics that capture both translation quality and editing behavior, supporting future APE research and deployment.

Abstract

Automatic post-editing (APE) aims to correct errors in machine-translated text, enhancing translation quality, while reducing the need for human intervention. Despite advances in neural machine translation (NMT), the development of effective APE systems has been hindered by the lack of large-scale multilingual datasets specifically tailored to NMT outputs. To address this gap, we present and release LangMark, a new human-annotated multilingual APE dataset for English translation to seven languages: Brazilian Portuguese, French, German, Italian, Japanese, Russian, and Spanish. The dataset has 206,983 triplets, with each triplet consisting of a source segment, its NMT output, and a human post-edited translation. Annotated by expert human linguists, our dataset offers both linguistic diversity and scale. Leveraging this dataset, we empirically show that Large Language Models (LLMs) with few-shot prompting can effectively perform APE, improving upon leading commercial and even proprietary machine translation systems. We believe that this new resource will facilitate the future development and evaluation of APE systems.

Figures (7)

• Figure 1: Example of a triplet in an automatic post-editing task.
• Figure 2: Two triplets from the LangMark dataset. These examples illustrate the nuanced nature of the required corrections. While the translations provided by the NMT engine are not inherently incorrect, they are inappropriate given the context of the source material (official marketing documents). For example, "our people" was misinterpreted as "our nation/community" in Spanish, and "pitch" was translated based on the meaning of "tar" in German instead of its intended meaning in a business context.
• Figure 3: Dataset statistics: (a) distribution of word counts for source segments, (b) lexical diversity measured using window-based TTR across languages, and (c) relative frequency of MQM error types in the pre-translations that need correction.
• Figure 4: Structure of the few-shot prompting format used for LLMs. If the model's API does not support a system prompt we simply prepend it to the user prompt.
• Figure 5: Normalized number of edits made by each model on the NMT output. Note that all models made significantly fewer edits than the human baseline. This indicates that there is still considerable room for improvement