TinyStyler: Efficient Few-Shot Text Style Transfer with Authorship Embeddings

TL;DR

TinyStyler is introduced, a lightweight but effective approach, which leverages a small language model and pre-trained authorship embeddings to perform efficient, few-shot text style transfer and outperforms recent controllable text generation methods.

Abstract

The goal of text style transfer is to transform the style of texts while preserving their original meaning, often with only a few examples of the target style. Existing style transfer methods generally rely on the few-shot capabilities of large language models or on complex controllable text generation approaches that are inefficient and underperform on fluency metrics. We introduce TinyStyler, a lightweight but effective approach, which leverages a small language model (800M params) and pre-trained authorship embeddings to perform efficient, few-shot text style transfer. We evaluate on the challenging task of authorship style transfer and find TinyStyler outperforms strong approaches such as GPT-4. We also evaluate TinyStyler's ability to perform text attribute style transfer (formal $\leftrightarrow$ informal) with automatic and human evaluations and find that the approach outperforms recent controllable text generation methods. Our model has been made publicly available at https://huggingface.co/tinystyler/tinystyler .

Figures (4)

• Figure 1: TinyStyler uses authorship embeddings from examples of the target style and conditions on these to rewrite source texts to match the target style. We replace expletives above with '$\ast$'.
• Figure 2: Step 1) We train a model to reconstruct texts from their paraphrases following krishna2020reformulating, however, we only train a single model for all styles. To do this, we condition reconstruction on pre-trained authorship embeddings. Step 2) We generate style transfer pairs by transforming Reddit posts from a source author by conditioning generation on authorship embeddings from a different Reddit author. We filter low-quality style transfer pairs automatically using meaning preservation and stylistic similarity metrics. Step 3) We self-distill our model on the remaining high-quality pairs to improve the consistency of our approach and remove the reliance on a separate, external paraphrasing model.
• Figure 3: TinyStyler affords control over the strength of style transfer by interpolating between the source and target styles in Style embedding space. The effect on style transfer metrics for different degrees of interpolation are visualized using GYAFC.
• Figure 4: We transform a formal text in GYAFC by interpolating (0% to 100%) towards the average embedding of the informal texts with TinyStyler. We visualize the outputs alongside samples from the corpus using a t-SNE projection JMLR:v9:vandermaaten08a. Texts are embedded with style embeddings.