Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Advancing Semi-Supervised Learning for Automatic Post-Editing: Data-Synthesis by Mask-Infilling with Erroneous Terms

Wonkee Lee, Seong-Hwan Heo, Jong-Hyeok Lee

TL;DR

To address data scarcity in automatic post-editing, the paper proposes data-synthesis methods that generate realistically corrupted MT outputs by masking and infilling references (MLM noising) and by selectively merging different synthetic data sources (selective corpus interleaving). The MLM noising approach trains a cross-lingual masked language model to predict plausible erroneous MT tokens given a source sentence and a masked reference, producing diverse synthetic triplets. Experimental results on the WMT'18 EN-DE APE data show that MLM noising with interleaving outperforms Trans, BT, and Rand baselines in TER and BLEU, with best gains when mixing roughly equally (lambda around 2). The findings demonstrate that aligning synthetic data to gold-error statistics and careful sample selection can significantly improve semi-supervised APE performance, suggesting directions for future MLM-based data augmentation techniques.

Abstract

Semi-supervised learning that leverages synthetic data for training has been widely adopted for developing automatic post-editing (APE) models due to the lack of training data. With this aim, we focus on data-synthesis methods to create high-quality synthetic data. Given that APE takes as input a machine-translation result that might include errors, we present a data-synthesis method by which the resulting synthetic data mimic the translation errors found in actual data. We introduce a noising-based data-synthesis method by adapting the masked language model approach, generating a noisy text from a clean text by infilling masked tokens with erroneous tokens. Moreover, we propose selective corpus interleaving that combines two separate synthetic datasets by taking only the advantageous samples to enhance the quality of the synthetic data further. Experimental results show that using the synthetic data created by our approach results in significantly better APE performance than other synthetic data created by existing methods.

Advancing Semi-Supervised Learning for Automatic Post-Editing: Data-Synthesis by Mask-Infilling with Erroneous Terms

TL;DR

To address data scarcity in automatic post-editing, the paper proposes data-synthesis methods that generate realistically corrupted MT outputs by masking and infilling references (MLM noising) and by selectively merging different synthetic data sources (selective corpus interleaving). The MLM noising approach trains a cross-lingual masked language model to predict plausible erroneous MT tokens given a source sentence and a masked reference, producing diverse synthetic triplets. Experimental results on the WMT'18 EN-DE APE data show that MLM noising with interleaving outperforms Trans, BT, and Rand baselines in TER and BLEU, with best gains when mixing roughly equally (lambda around 2). The findings demonstrate that aligning synthetic data to gold-error statistics and careful sample selection can significantly improve semi-supervised APE performance, suggesting directions for future MLM-based data augmentation techniques.

Abstract

Semi-supervised learning that leverages synthetic data for training has been widely adopted for developing automatic post-editing (APE) models due to the lack of training data. With this aim, we focus on data-synthesis methods to create high-quality synthetic data. Given that APE takes as input a machine-translation result that might include errors, we present a data-synthesis method by which the resulting synthetic data mimic the translation errors found in actual data. We introduce a noising-based data-synthesis method by adapting the masked language model approach, generating a noisy text from a clean text by infilling masked tokens with erroneous tokens. Moreover, we propose selective corpus interleaving that combines two separate synthetic datasets by taking only the advantageous samples to enhance the quality of the synthetic data further. Experimental results show that using the synthetic data created by our approach results in significantly better APE performance than other synthetic data created by existing methods.
Paper Structure (28 sections, 4 equations, 6 figures, 3 tables, 2 algorithms)

This paper contains 28 sections, 4 equations, 6 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Problem Statement
  4. Existing Data-Synthesis Methods
  5. Translation Approach (Trans)
  6. Back-Translation Approach (BT)
  7. Random Noising Approach (Rand)
  8. Approach
  9. Overview and Motivation
  10. MLM Noising
  11. Architecture and Objective
  12. Training Data Configuration
  13. Inference: Synthetic APE Data Construction
  14. Selective Corpus Interleaving
  15. Experiments
  16. ...and 13 more sections

Figures (6)

  • Figure 1: Overview of the APE process for an English-German translation. Bold highlights indicate erroneous words in $mt$ and post-edited words in $pe$.
  • Figure 2: Categorical distributions of gold (WMT) and synthetic (Trans) APE data, representing the proportion of samples belonging to a particular interval of translation error rate (TER) [%], a similarity measure based on the edit distance between $mt$ and its target (i.e., $pe$ or $ref$).
  • Figure 3: Overall architecture of our MLM noising model.
  • Figure 4: Schematic architecture of the concat-based APE model, which is among the fundamental APE models and exhibits outstanding performance.
  • Figure 5: Effect of the selective corpus interleaving when applied to other existing synthetic APE data. The colored plots represent the best performance for each data.
  • ...and 1 more figures