SynCED-EnDe 2025: A Synthetic and Curated English - German Dataset for Critical Error Detection in Machine Translation

TL;DR

SynCED-EnDe introduces a balanced, temporally fresh English-German Critical Error Detection dataset with 1,000 gold and 8,000 silver examples drawn from 2024–2025 sources. It enriches evaluation with explicit error subclasses, trigger flags, and five auxiliary judgments, enabling risk and intricacy scoring ($Risk$ and $Intricacy$). Baseline experiments with XLM-R show substantial improvements over the prior WMT21 CED benchmark due to balanced labels and refined annotations, demonstrating learnability and stability across encoder sizes. The dataset complements existing WMT resources, supporting safer MT deployment in information retrieval and conversational AI, and is released publicly to support reproducible research; future work includes expanding to more languages, domains, and semi-automatic labeling pipelines.

Abstract

Critical Error Detection (CED) in machine translation aims to determine whether a translation is safe to use or contains unacceptable deviations in meaning. While the WMT21 English-German CED dataset provided the first benchmark, it is limited in scale, label balance, domain coverage, and temporal freshness. We present SynCED-EnDe, a new resource consisting of 1,000 gold-labeled and 8,000 silver-labeled sentence pairs, balanced 50/50 between error and non-error cases. SynCED-EnDe draws from diverse 2024-2025 sources (StackExchange, GOV.UK) and introduces explicit error subclasses, structured trigger flags, and fine-grained auxiliary judgments (obviousness, severity, localization complexity, contextual dependency, adequacy deviation). These enrichments enable systematic analyses of error risk and intricacy beyond binary detection. The dataset is permanently hosted on GitHub and Hugging Face, accompanied by documentation, annotation guidelines, and baseline scripts. Benchmark experiments with XLM-R and related encoders show substantial performance gains over WMT21 due to balanced labels and refined annotations. We envision SynCED-EnDe as a community resource to advance safe deployment of MT in information retrieval and conversational assistants, particularly in emerging contexts such as wearable AI devices.

Figures (4)

• Figure 1: SynCED-EnDe pipeline: sources from 2024-2025 (StackExchange, GOV.UK) are cleaned and translated (EN$\rightarrow$DE). Controlled errors are injected with prompts (lexical, numerical, negation, toxicity), and labels are refined via prompt-based rechecking: three LLM rounds for training, three LLM rounds + manual correction for evaluation.
• Figure 2: Evaluation dimensions in SynCED-EnDe (evaluation set). Balanced risk and intricacy distributions with meaningful separation of err/not, covering a wide range of subtle and severe cases.
• Figure 3: Evaluation dimensions in WMT21. Skewed distributions with many obvious or extreme cases, and weaker separation between err and not.
• Figure 4: Correlation heatmaps of evaluation dimensions. Left: SynCED-EnDe shows consistent and interpretable correlations between adequacy, severity, risk, and intricacy, with obviousness largely independent. Right: WMT21 shows mixed or negative correlations, indicating less consistent annotation patterns.