Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Style-Specific Neurons for Steering LLMs in Text Style Transfer

Wen Lai, Viktor Hangya, Alexander Fraser

Abstract

Text style transfer (TST) aims to modify the style of a text without altering its original meaning. Large language models (LLMs) demonstrate superior performance across multiple tasks, including TST. However, in zero-shot setups, they tend to directly copy a significant portion of the input text to the output without effectively changing its style. To enhance the stylistic variety and fluency of the text, we present sNeuron-TST, a novel approach for steering LLMs using style-specific neurons in TST. Specifically, we identify neurons associated with the source and target styles and deactivate source-style-only neurons to give target-style words a higher probability, aiming to enhance the stylistic diversity of the generated text. However, we find that this deactivation negatively impacts the fluency of the generated text, which we address by proposing an improved contrastive decoding method that accounts for rapid token probability shifts across layers caused by deactivated source-style neurons. Empirical experiments demonstrate the effectiveness of the proposed method on six benchmarks, encompassing formality, toxicity, politics, politeness, authorship, and sentiment.

Style-Specific Neurons for Steering LLMs in Text Style Transfer

Abstract

Text style transfer (TST) aims to modify the style of a text without altering its original meaning. Large language models (LLMs) demonstrate superior performance across multiple tasks, including TST. However, in zero-shot setups, they tend to directly copy a significant portion of the input text to the output without effectively changing its style. To enhance the stylistic variety and fluency of the text, we present sNeuron-TST, a novel approach for steering LLMs using style-specific neurons in TST. Specifically, we identify neurons associated with the source and target styles and deactivate source-style-only neurons to give target-style words a higher probability, aiming to enhance the stylistic diversity of the generated text. However, we find that this deactivation negatively impacts the fluency of the generated text, which we address by proposing an improved contrastive decoding method that accounts for rapid token probability shifts across layers caused by deactivated source-style neurons. Empirical experiments demonstrate the effectiveness of the proposed method on six benchmarks, encompassing formality, toxicity, politics, politeness, authorship, and sentiment.
Paper Structure (32 sections, 5 equations, 4 figures, 12 tables)

This paper contains 32 sections, 5 equations, 4 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Identifying Style-Specific Neurons
  5. Neurons in LLMs
  6. Neuron Selection
  7. Deactivating Source Style Neurons
  8. Contrastive Decoding for TST
  9. Dola
  10. Our adaptation to TST
  11. Candidate layer selection.
  12. Next-token prediction.
  13. Experiments
  14. Datasets
  15. Baselines
  16. ...and 17 more sections

Figures (4)

  • Figure 1: Method overview. The whole framework consists of three parts: identifying style-specific neurons, deactivating source style neurons, and decoding by contrasting style layer. The histogram represents the probability distribution of each word across different layers. When source style neurons are deactivated, LLMs tend to generate all target-style words, such as "Neither" and "poor". By employing contrastive decoding, LLMs take fluency into account and reduce the probability of generating "poor".
  • Figure 2: Overlap statistics of style-specific neurons identified using the method of tang2024language on six benchmarks.
  • Figure 3: Statistics of the number of style-specific neurons in each layer in LLaMA-3 on formality and toxicity benchmarks.
  • Figure 4: Copy Ratio on three selected TST tasks. Lower value indicates better performance of the model.