Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generation, Evaluation, and Explanation of Novelists' Styles with Single-Token Prompts

Mosab Rezaei, Mina Rajaei Moghadam, Abdul Rahman Shaikh, Hamed Alhoori, Reva Freedman

TL;DR

This work tackles stylometric generation under non-parallel data by employing single-token prompts to imitate five 19th-century authors. It couples GPT-Neo fine-tuning (FFT and LoRA) with a DeBERTa V3-Large classifier for evaluation and uses Attention Enrichment and Integrated Gradients to explain the generation process. The results show that FFT yields higher stylistic fidelity than LoRA, and AI-based evaluation aligns well with intended styles, with robust token-level cues identified for each author. The approach offers a scalable, interpretable toolkit for digital libraries and literary analysis, enabling both preservation and creative exploration of authorial styles at scale.

Abstract

Recent advances in large language models have created new opportunities for stylometry, the study of writing styles and authorship. Two challenges, however, remain central: training generative models when no paired data exist, and evaluating stylistic text without relying only on human judgment. In this work, we present a framework for both generating and evaluating sentences in the style of 19th-century novelists. Large language models are fine-tuned with minimal, single-token prompts to produce text in the voices of authors such as Dickens, Austen, Twain, Alcott, and Melville. To assess these generative models, we employ a transformer-based detector trained on authentic sentences, using it both as a classifier and as a tool for stylistic explanation. We complement this with syntactic comparisons and explainable AI methods, including attention-based and gradient-based analyses, to identify the linguistic cues that drive stylistic imitation. Our findings show that the generated text reflects the authors' distinctive patterns and that AI-based evaluation offers a reliable alternative to human assessment. All artifacts of this work are published online.

Generation, Evaluation, and Explanation of Novelists' Styles with Single-Token Prompts

TL;DR

This work tackles stylometric generation under non-parallel data by employing single-token prompts to imitate five 19th-century authors. It couples GPT-Neo fine-tuning (FFT and LoRA) with a DeBERTa V3-Large classifier for evaluation and uses Attention Enrichment and Integrated Gradients to explain the generation process. The results show that FFT yields higher stylistic fidelity than LoRA, and AI-based evaluation aligns well with intended styles, with robust token-level cues identified for each author. The approach offers a scalable, interpretable toolkit for digital libraries and literary analysis, enabling both preservation and creative exploration of authorial styles at scale.

Abstract

Recent advances in large language models have created new opportunities for stylometry, the study of writing styles and authorship. Two challenges, however, remain central: training generative models when no paired data exist, and evaluating stylistic text without relying only on human judgment. In this work, we present a framework for both generating and evaluating sentences in the style of 19th-century novelists. Large language models are fine-tuned with minimal, single-token prompts to produce text in the voices of authors such as Dickens, Austen, Twain, Alcott, and Melville. To assess these generative models, we employ a transformer-based detector trained on authentic sentences, using it both as a classifier and as a tool for stylistic explanation. We complement this with syntactic comparisons and explainable AI methods, including attention-based and gradient-based analyses, to identify the linguistic cues that drive stylistic imitation. Our findings show that the generated text reflects the authors' distinctive patterns and that AI-based evaluation offers a reliable alternative to human assessment. All artifacts of this work are published online.

Paper Structure

This paper contains 12 sections, 3 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Procedure
  4. Dataset
  5. Generation Path
  6. Evaluation-Explanation Path
  7. Results
  8. Generation
  9. Evaluation
  10. Explanation
  11. Limitation and Future Work
  12. Conclusion

Figures (5)

  • Figure 1: The outline of the framework.
  • Figure 2: Evaluating the writing style of generated sentences. Left: Both generator models, FFT and LoRA. Right: Only the FFT results
  • Figure 3: Histogram of the longest path in parse trees, percentage of prepositional phrases, and number of words in real and generated sentences.
  • Figure 4: Integrated Gradients (IG): Five examples of the tag identifier effect on the next token prediction. Generated sentences are: < 0> "Aha!" said the spy, with a long draggled breath. < 1> I am the happiest creature in the world. < 2> The place has been picketed since the late war. < 4> The Commodore, like the Emperor of foreign nations, is a ...
  • Figure 5: Token-Level Attributions with Integrated Gradients.