Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AMPLIFY:Attention-based Mixup for Performance Improvement and Label Smoothing in Transformer

Leixin Yang, Yu Xiang

TL;DR

This method uses the attention mechanism of Transformer itself to reduce the influence of noises and aberrant values in the original samples on the prediction results, without increasing additional trainable parameters, and the computational cost is very low, thereby avoiding the problem of high resource consumption in common mixup methods such as Sentence Mixup.

Abstract

Mixup is an effective data augmentation method that generates new augmented samples by aggregating linear combinations of different original samples. However, if there are noises or aberrant features in the original samples, Mixup may propagate them to the augmented samples, leading to over-sensitivity of the model to these outliers . To solve this problem, this paper proposes a new Mixup method called AMPLIFY. This method uses the Attention mechanism of Transformer itself to reduce the influence of noises and aberrant values in the original samples on the prediction results, without increasing additional trainable parameters, and the computational cost is very low, thereby avoiding the problem of high resource consumption in common Mixup methods such as Sentence Mixup . The experimental results show that, under a smaller computational resource cost, AMPLIFY outperforms other Mixup methods in text classification tasks on 7 benchmark datasets, providing new ideas and new ways to further improve the performance of pre-trained models based on the Attention mechanism, such as BERT, ALBERT, RoBERTa, and GPT. Our code can be obtained at https://github.com/kiwi-lilo/AMPLIFY.

AMPLIFY:Attention-based Mixup for Performance Improvement and Label Smoothing in Transformer

TL;DR

This method uses the attention mechanism of Transformer itself to reduce the influence of noises and aberrant values in the original samples on the prediction results, without increasing additional trainable parameters, and the computational cost is very low, thereby avoiding the problem of high resource consumption in common mixup methods such as Sentence Mixup.

Abstract

Mixup is an effective data augmentation method that generates new augmented samples by aggregating linear combinations of different original samples. However, if there are noises or aberrant features in the original samples, Mixup may propagate them to the augmented samples, leading to over-sensitivity of the model to these outliers . To solve this problem, this paper proposes a new Mixup method called AMPLIFY. This method uses the Attention mechanism of Transformer itself to reduce the influence of noises and aberrant values in the original samples on the prediction results, without increasing additional trainable parameters, and the computational cost is very low, thereby avoiding the problem of high resource consumption in common Mixup methods such as Sentence Mixup . The experimental results show that, under a smaller computational resource cost, AMPLIFY outperforms other Mixup methods in text classification tasks on 7 benchmark datasets, providing new ideas and new ways to further improve the performance of pre-trained models based on the Attention mechanism, such as BERT, ALBERT, RoBERTa, and GPT. Our code can be obtained at https://github.com/kiwi-lilo/AMPLIFY.
Paper Structure (18 sections, 1 equation, 7 figures, 5 tables, 1 algorithm)

This paper contains 18 sections, 1 equation, 7 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Experiments
  5. Results
  6. Conclusion and Future Work
  7. Competing Interests
  8. Author Contributions
  9. Data Availability Statement

Figures (7)

  • Figure 1: A schematic of AMPLIFY. In each encoder block of the Transformer, the forward-propagated input data is duplicated and re-ordered according to the label-mixing order after obtaining the results of the Muti-Head Attention, and then Mixup operation is performed. No changes are made to other network structures. Similarly, each decoder block can also perform the same operation.
  • Figure 3: The figure presents a heatmap of the attention output matrices comparing our method with EmbedMix, SentenceMix, TMix, and the baseline. Each element in the matrix represents the p-value obtained from conducting a t-test between our method and the other methods. Different color distributions are used to differentiate the magnitude of p-values.
  • Figure 4: The cross-entropy loss values for four Mixup methods, EmbedMix, TMix, AMPLIFY, and SentenceMix, on the MRPC dataset in the first 12k training iterations.
  • Figure 5: The influence of the fourth MHA layer of the model on the same text sequence after undergoing AMPLIFY and EmbedMix operations, respectively. The left side of the figure represents the word being updated, while the right side represents the word being processed. The lines in the figure represent the semantic correlations between words, and the color depth reflects the weight of attention obtained from the correlation. The text sequence consists of two sentences "the rabbit quickly hopped" and "the turtle slowly crawled", with [SEP] being a special token used to separate the two sentences and [CLS] being a special token used to classify the text sequence.
  • Figure 6: The influence of the fourth MHA layer of the model, after undergoing AMPLIFY and EmbedMix operations, respectively, on a specific word "rabbit".
  • ...and 2 more figures