Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unsupervised Text Style Transfer via LLMs and Attention Masking with Multi-way Interactions

Lei Pan, Yunshi Lan, Yang Li, Weining Qian

TL;DR

This work tackles unsupervised text style transfer by integrating attention masking with large language models (LLMs) through four interaction modes. It develops pipeline-based approaches (prompt-then-AM and AM-then-prompt), knowledge distillation from LLMs to an attention masking model, and in-context learning with demonstrations to guide generation. Empirical results on Yelp-clean and Amazon-clean show that multi-way interactions improve style strength, content preservation, and fluency, with prompting followed by attention masking achieving state-of-the-art mean performance, even against supervised baselines. The findings underscore the practical potential of combining LLM flexibility with controllable, mask-based edits for effective unsupervised style transfer, while acknowledging limitations such as hallucinations and sensitivity to prompting choices.

Abstract

Unsupervised Text Style Transfer (UTST) has emerged as a critical task within the domain of Natural Language Processing (NLP), aiming to transfer one stylistic aspect of a sentence into another style without changing its semantics, syntax, or other attributes. This task is especially challenging given the intrinsic lack of parallel text pairings. Among existing methods for UTST tasks, attention masking approach and Large Language Models (LLMs) are deemed as two pioneering methods. However, they have shortcomings in generating unsmooth sentences and changing the original contents, respectively. In this paper, we investigate if we can combine these two methods effectively. We propose four ways of interactions, that are pipeline framework with tuned orders; knowledge distillation from LLMs to attention masking model; in-context learning with constructed parallel examples. We empirically show these multi-way interactions can improve the baselines in certain perspective of style strength, content preservation and text fluency. Experiments also demonstrate that simply conducting prompting followed by attention masking-based revision can consistently surpass the other systems, including supervised text style transfer systems. On Yelp-clean and Amazon-clean datasets, it improves the previously best mean metric by 0.5 and 3.0 absolute percentages respectively, and achieves new SOTA results.

Unsupervised Text Style Transfer via LLMs and Attention Masking with Multi-way Interactions

TL;DR

This work tackles unsupervised text style transfer by integrating attention masking with large language models (LLMs) through four interaction modes. It develops pipeline-based approaches (prompt-then-AM and AM-then-prompt), knowledge distillation from LLMs to an attention masking model, and in-context learning with demonstrations to guide generation. Empirical results on Yelp-clean and Amazon-clean show that multi-way interactions improve style strength, content preservation, and fluency, with prompting followed by attention masking achieving state-of-the-art mean performance, even against supervised baselines. The findings underscore the practical potential of combining LLM flexibility with controllable, mask-based edits for effective unsupervised style transfer, while acknowledging limitations such as hallucinations and sensitivity to prompting choices.

Abstract

Unsupervised Text Style Transfer (UTST) has emerged as a critical task within the domain of Natural Language Processing (NLP), aiming to transfer one stylistic aspect of a sentence into another style without changing its semantics, syntax, or other attributes. This task is especially challenging given the intrinsic lack of parallel text pairings. Among existing methods for UTST tasks, attention masking approach and Large Language Models (LLMs) are deemed as two pioneering methods. However, they have shortcomings in generating unsmooth sentences and changing the original contents, respectively. In this paper, we investigate if we can combine these two methods effectively. We propose four ways of interactions, that are pipeline framework with tuned orders; knowledge distillation from LLMs to attention masking model; in-context learning with constructed parallel examples. We empirically show these multi-way interactions can improve the baselines in certain perspective of style strength, content preservation and text fluency. Experiments also demonstrate that simply conducting prompting followed by attention masking-based revision can consistently surpass the other systems, including supervised text style transfer systems. On Yelp-clean and Amazon-clean datasets, it improves the previously best mean metric by 0.5 and 3.0 absolute percentages respectively, and achieves new SOTA results.
Paper Structure (24 sections, 6 equations, 3 figures, 6 tables)

This paper contains 24 sections, 6 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Task Definition
  4. Baseline UTST System
  5. Attention Masking Method
  6. LLM-based Method
  7. Pipeline UTST System
  8. Prompt-then-AM
  9. AM-then-prompt
  10. Using LLM Outputs as Signals
  11. Using AM Outputs as Demonstrations
  12. Experimental Setup
  13. Dataset
  14. Comparable Methods
  15. Evaluation metrics
  16. ...and 9 more sections

Figures (3)

  • Figure 1: UTST system via LLMs and Attention Masking (AM) with four-way interactions: Prompt-then-AM, AM-then-prompt, knowledge distillation using LLM outputs as signals, and in-context learning using AM outputs as demonstrations. The details of attention masking module and LLM-based module are displayed at the left side, where the black arrow denotes propagation and the red arrow denotes back-propagation.
  • Figure 2: ACC and s-sBLEU of Prompt-then-AM on Yelp-clean (N$\rightarrow$P) with increasing $\alpha$.
  • Figure 3: ACC and PPL of Prompt-then-AM on Yelp-clean (N$\rightarrow$P) with various $\alpha$.