Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On-the-fly Denoising for Data Augmentation in Natural Language Understanding

Tianqing Fang, Wenxuan Zhou, Fangyu Liu, Hongming Zhang, Yangqiu Song, Muhao Chen

TL;DR

The paper tackles noise in data augmentation for NLP by introducing On-the-fly Denoising for Data Augmentation (ODDA). ODDA jointly leverages an organic distillation teacher trained on clean original data to softly supervise augmented examples and adds a dropout-based self-regularization loss to mitigate overfitting to noisy labels, without discarding augmented data. The approach demonstrates consistent improvements over filtering, re-weighting, and consistency-training baselines on both token-level augmentation (EDA) for text classification and sentence-level augmentation (G-DAUG) for commonsense QA, including few-shot and full-shot setups. The method is data-efficient, generalizable across augmentation types, and provides a practical denoising mechanism for improving NLP model robustness in low-resource settings.

Abstract

Data Augmentation (DA) is frequently used to provide additional training data without extra human annotation automatically. However, data augmentation may introduce noisy data that impairs training. To guarantee the quality of augmented data, existing methods either assume no noise exists in the augmented data and adopt consistency training or use simple heuristics such as training loss and diversity constraints to filter out "noisy" data. However, those filtered examples may still contain useful information, and dropping them completely causes a loss of supervision signals. In this paper, based on the assumption that the original dataset is cleaner than the augmented data, we propose an on-the-fly denoising technique for data augmentation that learns from soft augmented labels provided by an organic teacher model trained on the cleaner original data. To further prevent overfitting on noisy labels, a simple self-regularization module is applied to force the model prediction to be consistent across two distinct dropouts. Our method can be applied to general augmentation techniques and consistently improve the performance on both text classification and question-answering tasks.

On-the-fly Denoising for Data Augmentation in Natural Language Understanding

TL;DR

The paper tackles noise in data augmentation for NLP by introducing On-the-fly Denoising for Data Augmentation (ODDA). ODDA jointly leverages an organic distillation teacher trained on clean original data to softly supervise augmented examples and adds a dropout-based self-regularization loss to mitigate overfitting to noisy labels, without discarding augmented data. The approach demonstrates consistent improvements over filtering, re-weighting, and consistency-training baselines on both token-level augmentation (EDA) for text classification and sentence-level augmentation (G-DAUG) for commonsense QA, including few-shot and full-shot setups. The method is data-efficient, generalizable across augmentation types, and provides a practical denoising mechanism for improving NLP model robustness in low-resource settings.

Abstract

Data Augmentation (DA) is frequently used to provide additional training data without extra human annotation automatically. However, data augmentation may introduce noisy data that impairs training. To guarantee the quality of augmented data, existing methods either assume no noise exists in the augmented data and adopt consistency training or use simple heuristics such as training loss and diversity constraints to filter out "noisy" data. However, those filtered examples may still contain useful information, and dropping them completely causes a loss of supervision signals. In this paper, based on the assumption that the original dataset is cleaner than the augmented data, we propose an on-the-fly denoising technique for data augmentation that learns from soft augmented labels provided by an organic teacher model trained on the cleaner original data. To further prevent overfitting on noisy labels, a simple self-regularization module is applied to force the model prediction to be consistent across two distinct dropouts. Our method can be applied to general augmentation techniques and consistently improve the performance on both text classification and question-answering tasks.
Paper Structure (28 sections, 8 equations, 3 figures, 8 tables, 1 algorithm)

This paper contains 28 sections, 8 equations, 3 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Data Augmentation and Filtering
  4. Learning with Noisy Labels
  5. Method
  6. Problem Formulation
  7. On-the-fly Denoising
  8. Organic Distillation (OD).
  9. Self-Regularization (SR).
  10. Joint Training
  11. Experiments
  12. Text Classification
  13. Setup.
  14. Baselines.
  15. Results and Analysis.
  16. ...and 13 more sections

Figures (3)

  • Figure 1: An example in a sentiment classification task about the noise brought by text-editing data augmentation. The noisy augmented text has the probability of being a "positive" attitude due to the removal of "not".
  • Figure 2: Overview of our ODDA framework.
  • Figure 3: (1) The effect of OD temperature $\tau$ on the classification performance for AGNews dataset. (2) The effect of SR coefficient $\alpha$ on the classification performance for TREC dataset.