Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training

Khoi M. Le, Trinh Pham, Tho Quan, Anh Tuan Luu

TL;DR

The first unsupervised multilingual paraphrasing model, LAMPAT (Low-rank Adaptation for Multilingual Paraphrasing using Adversarial Training), is proposed, by which monolingual dataset is sufficient enough to generate a human-like and diverse sentence.

Abstract

Paraphrases are texts that convey the same meaning while using different words or sentence structures. It can be used as an automatic data augmentation tool for many Natural Language Processing tasks, especially when dealing with low-resource languages, where data shortage is a significant problem. To generate a paraphrase in multilingual settings, previous studies have leveraged the knowledge from the machine translation field, i.e., forming a paraphrase through zero-shot machine translation in the same language. Despite good performance on human evaluation, those methods still require parallel translation datasets, thus making them inapplicable to languages that do not have parallel corpora. To mitigate that problem, we proposed the first unsupervised multilingual paraphrasing model, LAMPAT ($\textbf{L}$ow-rank $\textbf{A}$daptation for $\textbf{M}$ultilingual $\textbf{P}$araphrasing using $\textbf{A}$dversarial $\textbf{T}$raining), by which monolingual dataset is sufficient enough to generate a human-like and diverse sentence. Throughout the experiments, we found out that our method not only works well for English but can generalize on unseen languages as well. Data and code are available at https://github.com/VinAIResearch/LAMPAT.

LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training

TL;DR

The first unsupervised multilingual paraphrasing model, LAMPAT (Low-rank Adaptation for Multilingual Paraphrasing using Adversarial Training), is proposed, by which monolingual dataset is sufficient enough to generate a human-like and diverse sentence.

Abstract

Paraphrases are texts that convey the same meaning while using different words or sentence structures. It can be used as an automatic data augmentation tool for many Natural Language Processing tasks, especially when dealing with low-resource languages, where data shortage is a significant problem. To generate a paraphrase in multilingual settings, previous studies have leveraged the knowledge from the machine translation field, i.e., forming a paraphrase through zero-shot machine translation in the same language. Despite good performance on human evaluation, those methods still require parallel translation datasets, thus making them inapplicable to languages that do not have parallel corpora. To mitigate that problem, we proposed the first unsupervised multilingual paraphrasing model, LAMPAT (ow-rank daptation for ultilingual araphrasing using dversarial raining), by which monolingual dataset is sufficient enough to generate a human-like and diverse sentence. Throughout the experiments, we found out that our method not only works well for English but can generalize on unseen languages as well. Data and code are available at https://github.com/VinAIResearch/LAMPAT.
Paper Structure (23 sections, 3 equations, 3 figures, 6 tables)

This paper contains 23 sections, 3 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Definition
  4. Synthetic Parallel Corpora
  5. Parameter-Efficient Fine-Tuning
  6. Virtual Adversarial Training
  7. Experimental Setup
  8. Primary Model
  9. Baseline Model
  10. Evaluation Dataset
  11. Automatic Evaluation
  12. Human Evaluation
  13. Results
  14. Main Results
  15. Human Evaluation Results
  16. ...and 8 more sections

Figures (3)

  • Figure 1: The training process of LAMPAT consists of multiple stages. Firstly, we create a synthetic parallel corpus using unsupervised monolingual data. Next, we utilize LoRA to effectively fine-tune our model. Finally, we obtain the self-supervised model LAMPAT through the utilization of Virtual Adversarial Training.
  • Figure 2: LAMPAT is showcased using actual inputs. Initially, an input text undergoes corruption by removing stopwords and shuffling. Then, a noise perturbation, denoted as $\delta$, is introduced into the text embedding to generate a paraphrase that exhibits lexical diversity. The transformer block is replicated $N$ times, with the Multi-Head Attention component decomposed into low-rank matrices for efficient fine-tuning. Lastly, LAMPAT is trained using virtual adversarial training, incorporating a two-component loss function: the reconstruction loss $\mathcal{L}_{rec}$ and the virtual adversarial regularizer $\mathcal{L}_{vadv}$.
  • Figure 3: The average ParaScore of each technique over 13 languages.