Learning to Generate Text in Arbitrary Writing Styles

TL;DR

The proposed approach (StyleMC) combines an author-adapted language model with sequence-level inference to improve stylistic consistency, and is found to be effective in a variety of conditions, including unconditional generation and style transfer.

Abstract

Prior work in style-controlled text generation has focused on tasks such as emulating the style of prolific literary authors, producing formal or informal text, and mitigating toxicity of generated text. Plentiful demonstrations of these styles are available, and as a result modern language models are often able to emulate them, either via prompting or discriminative control. However, in applications such as writing assistants, it is desirable for language models to produce text in an author-specific style on the basis of a potentially small writing sample. For example, someone writing in a particular dialect may prefer writing suggestions that retain the same dialect. We find that instruction-tuned language models can struggle to reproduce author-specific style demonstrated in a prompt. Instead, we propose to guide a language model to generate text in a target style using contrastively-trained representations that capture stylometric features. Our approach (StyleMC) combines an author-adapted language model with sequence-level inference to improve stylistic consistency, and is found to be effective in a variety of conditions, including unconditional generation and style transfer. Additionally, we find that the proposed approach can serve as an effective anonymization method, by editing a document to mask authorship while preserving the original meaning

Figures (3)

• Figure 1: An overview of StyleMC for style transfer. Using MCMC, we generate text based on a few samples of the target style, an author-specific fluency model (§\ref{['sec:ebm']}), and the original content. Our approach reproduces salient features of an author's style while preserving meaning. In the (real) example above, accepted samples exhibit characteristics of British English (yellow), matching the characteristics of the target style.
• Figure 2: Style performance for an increasing number of examples of a target style. We find that more examples result in better representations, which in turn improve decoding quality. Our proposed approach, including an ablation without sequence-level decoding (- EBM), significantly outperforms much larger models using prompting strategies.
• Figure 3: Percent of capitalized and punctuation characters in generated outputs. The decoding procedure is run on interpolated style vectors, where a weight of 0.0 is indicates a style vector capturing a nocaps or nopunct behavior, and a weight of 1.0 corresponds to a normal /r/wsb user.