Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting

TL;DR

This work tackles sentence rewriting with seq2seq models by isolating augmented data in a pre-training phase and gold data in a fine-tuning phase, mitigating the noise inherent in augmented data when trained jointly. It introduces back translation, feature discrimination, and multi-task transfer as augmentation strategies to enrich pre-training, validated on GEC and FST benchmarks where it achieves new state-of-the-art results. The approach demonstrates strong gains across CoNLL-2014, JFLEG, and GYAFC, with evidence from both automatic metrics and human evaluation, and shows promise for generalization to other seq2seq tasks. Overall, the two-phase PT&FT paradigm provides a robust framework for leveraging data augmentation in sentence rewriting with improved reliability and performance.

Abstract

We study sequence-to-sequence (seq2seq) pre-training with data augmentation for sentence rewriting. Instead of training a seq2seq model with gold training data and augmented data simultaneously, we separate them to train in different phases: pre-training with the augmented data and fine-tuning with the gold data. We also introduce multiple data augmentation methods to help model pre-training for sentence rewriting. We evaluate our approach in two typical well-defined sentence rewriting tasks: Grammatical Error Correction (GEC) and Formality Style Transfer (FST). Experiments demonstrate our approach can better utilize augmented data without hurting the model's trust in gold data and further improve the model's performance with our proposed data augmentation methods. Our approach substantially advances the state-of-the-art results in well-recognized sentence rewriting benchmarks over both GEC and FST. Specifically, it pushes the CoNLL-2014 benchmark's $F_{0.5}$ score and JFLEG Test GLEU score to 62.61 and 63.54 in the restricted training setting, 66.77 and 65.22 respectively in the unrestricted setting, and advances GYAFC benchmark's BLEU to 74.24 (2.23 absolute improvement) in E&M domain and 77.97 (2.64 absolute improvement) in F&R domain.

Figures (4)

• Figure 1: An augmented sentence pair generated through back-translation (BT) for GEC. Though it includes useful rewriting knowledge (the underlined text) for GEC, it additionally introduces undesirable edit (the bold text) which may lead the model to learn to rewrite the content that should not be edited.
• Figure 2: An example that Formality Style Transfer (FST) benefits from data augmented via feature discrimination (F-Dis) and multi-task transfer (M-Task). F-Dis identifies useful sentence pairs whose target's formality score (the numbers in the parentheses) is higher than the source, from paraphrase sentences generated by cross-lingual MT, while M-Task utilizes training data for GEC to help formality improvement.
• Figure 3: Comparison between (a) Simultaneous Training and (b) Pre-training & Fine-tuning framework.
• Figure 4: Fluency discrimination for GEC. The paraphrased sentences are generated by a back translation model trained on GEC parallel data. The scores right after the sentences are their fluency scores. The fluency discriminator only chooses the sentence whose fluency score is lower than the correct sentence and pairs them (by the red dashed arrow) as augmented data.