StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples

TL;DR

StyleDistance addresses content leakage in style representations by training content-independent embeddings with synthetic parallel data generated by an LLM. It introduces SynthStel, a dataset of near-exact paraphrases with controlled 40 style features, and trains a RoBERTa-based encoder with a triplet loss $L_t(\theta) = \sum_{(a,p,n)\in \mathcal{D}} [ \left\| f_\theta(a) - f_\theta(p) \right\|_2^2 - \left\| f_\theta(a) - f_\theta(n) \right\|_2^2 + \alpha \right]_+$; optimizing to separate style from content. The results show improved content-independence on STEL and STEL-or-Content benchmarks, with strong generalization to unseen style features and competitive or superior performance on downstream tasks like authorship verification and automatic style transfer evaluation. The authors release SynthStel, StyleDistance models, and code to enable replication and application in style analysis tasks.

Abstract

Style representations aim to embed texts with similar writing styles closely and texts with different styles far apart, regardless of content. However, the contrastive triplets often used for training these representations may vary in both style and content, leading to potential content leakage in the representations. We introduce StyleDistance, a novel approach to training stronger content-independent style embeddings. We use a large language model to create a synthetic dataset of near-exact paraphrases with controlled style variations, and produce positive and negative examples across 40 distinct style features for precise contrastive learning. We assess the quality of our synthetic data and embeddings through human and automatic evaluations. StyleDistance enhances the content-independence of style embeddings, which generalize to real-world benchmarks and outperform leading style representations in downstream applications. Our model can be found at https://huggingface.co/StyleDistance/styledistance .

Figures (8)

• Figure 1: StyleDistance embeddings are trained using contrastive learning from synthetic parallel (positive and negative) examples representing 40 style features. The illustrated example is for the "Usage of Active Voice" feature.
• Figure 2: In the training triplets used by styleemb (left), an anchor text is paired with a positive instance written by the same author, and a negative instance written by a different author, assuming content can be controlled via subreddit/conversation metadata. However, this assumption can fail, leading to uncontrolled content as illustrated. In our dataset used to train StyleDistance (right), we control for both style and content.
• Figure 3: We generate synthetic parallel examples to train StyleDistance for a wide range of style features in seven linguistic and stylistic categories. Further details on these features can be found in Appendix \ref{['sec:appendix:stylefeatures']}.
• Figure 4: The annotation interface used for human annotation.
• Figure 6: A visualization of the STEL and STEL-or-Content task evaluation we describe in Section \ref{['sec:steleval']}.