Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Machine Translation Detection Using a Surrogate Multilingual Translation Model

Cristian García-Romero, Miquel Esplà-Gomis, Felipe Sánchez-Martínez

TL;DR

This work tackles the problem of identifying machine-translated content within parallel corpora, a key pre-processing step for training high-quality MT systems. It introduces SMaTD, which directly uses latent representations from a pre-trained surrogate multilingual MT model, specifically decoder-block states $h_{k,i}^{(d)}$ projected to $h^{(d')}$, to perform binary HT vs MT classification, with an optional SMaTD+LM extension that concatenates LM-derived features. Empirical results show that SMaTD consistently outperforms state-of-the-art baselines, with especially large gains for non-English language pairs and in zero-shot settings, and it generalizes well across languages and MT systems. The approach enables robust, language-agnostic MT-content filtering for multilingual corpora, contributing to better translation quality and more reliable MT datasets, while also offering insights into which model components carry discriminative MT-detection signals.

Abstract

Modern machine translation (MT) systems depend on large parallel corpora, often collected from the Internet. However, recent evidence indicates that (i) a substantial portion of these texts are machine-generated translations, and (ii) an overreliance on such synthetic content in training data can significantly degrade translation quality. As a result, filtering out non-human translations is becoming an essential pre-processing step in building high-quality MT systems. In this work, we propose a novel approach that directly exploits the internal representations of a surrogate multilingual MT model to distinguish between human and machine-translated sentences. Experimental results show that our method outperforms current state-of-the-art techniques, particularly for non-English language pairs, achieving gains of at least 5 percentage points of accuracy.

Automatic Machine Translation Detection Using a Surrogate Multilingual Translation Model

TL;DR

This work tackles the problem of identifying machine-translated content within parallel corpora, a key pre-processing step for training high-quality MT systems. It introduces SMaTD, which directly uses latent representations from a pre-trained surrogate multilingual MT model, specifically decoder-block states projected to , to perform binary HT vs MT classification, with an optional SMaTD+LM extension that concatenates LM-derived features. Empirical results show that SMaTD consistently outperforms state-of-the-art baselines, with especially large gains for non-English language pairs and in zero-shot settings, and it generalizes well across languages and MT systems. The approach enables robust, language-agnostic MT-content filtering for multilingual corpora, contributing to better translation quality and more reliable MT datasets, while also offering insights into which model components carry discriminative MT-detection signals.

Abstract

Modern machine translation (MT) systems depend on large parallel corpora, often collected from the Internet. However, recent evidence indicates that (i) a substantial portion of these texts are machine-generated translations, and (ii) an overreliance on such synthetic content in training data can significantly degrade translation quality. As a result, filtering out non-human translations is becoming an essential pre-processing step in building high-quality MT systems. In this work, we propose a novel approach that directly exploits the internal representations of a surrogate multilingual MT model to distinguish between human and machine-translated sentences. Experimental results show that our method outperforms current state-of-the-art techniques, particularly for non-English language pairs, achieving gains of at least 5 percentage points of accuracy.

Paper Structure

This paper contains 23 sections, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Combination with a Language Model
  4. Experimental Setting
  5. Datasets.
  6. Evaluation.
  7. Baselines.
  8. SMaTD training.
  9. SMaTD+LM training.
  10. Results and Discussion
  11. Surrogate MT Model Size and Layer Evaluation
  12. Main Results
  13. Cross-Evaluation
  14. Cross-MT.
  15. Cross-Lingual.
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Per-word perplexity for human and MT-generated translations (produced with MADLAD) from our English--German training set (see Sec. \ref{['section:experimental-setting']}). Perplexity is obtained with NLLB 3.3B. Similar trends are observed across language pairs and MT models.
  • Figure 2: Architecture of SMaTD/SMaTD+LM. Some elements (e.g., positional embeddings) have been omitted for clarity. See Sec. \ref{['section:approach']} for a detailed explanation.
  • Figure 3: Accuracy on the development set for DeepL (de--en) evaluated using three different sizes of the NLLB surrogate MT model. For each surrogate model, we evaluate the use of the hidden state of different decoder blocks as input to the classifier; note that the 600M model has fewer decoder blocks by design.