A Stylometric Application of Large Language Models

TL;DR

This work investigates whether large language models can capture author-specific writing styles by training separate GPT-2 models on individual authors and using cross-entropy loss on held-out texts as a stylometric distance. The key idea, predictive comparison, treats lower cross-entropy as closer stylistic similarity and enables both attribution and distance estimation, including a formal distance metric $d(i,j) = \tfrac{1}{2}\left(\overline{L_j(i)} + \overline{L_i(j)}\right)$. Empirically, models trained on the same author consistently outperform others, achieving perfect author discrimination across eight authors and successfully resolving the contested Oz book authorship; ablation studies show both content and function words contribute to the signature, while POS-only signals are weaker. The approach offers a simple, open-set-friendly framework for computational stylometry and literary attribution, though it faces limitations in scope, interpretability, and cross-domain robustness, with future work aimed at theory, efficiency, and broader applications.

Abstract

We show that large language models (LLMs) can be used to distinguish the writings of different authors. Specifically, an individual GPT-2 model, trained from scratch on the works of one author, will predict held-out text from that author more accurately than held-out text from other authors. We suggest that, in this way, a model trained on one author's works embodies the unique writing style of that author. We first demonstrate our approach on books written by eight different (known) authors. We also use this approach to confirm R. P. Thompson's authorship of the well-studied 15th book of the Oz series, originally attributed to F. L. Baum.

Figures (5)

• Figure 1: Cross-entropy loss across models and authors.A. Average cross-entropy loss on Training data and held-out test data from each author, plotted as a function of the number of training epochs. Each color denotes a model trained on a single author's work. Error ribbons denote bootstrap-estimated 95% confidence intervals over 10 random seeds. B. Cross-entropy loss assigned to held-out test data by each author's model ($x$-axis). Held-out test data is either from the same author (black) or from other authors (gray). Each dot denotes the average loss (across all 1024-token chunks) for a single random seed. See Supplementary Materials for analogous plots using models trained on only content words (Supp. Fig. 1), only function words (Supp. Fig. 2), and only parts of speech (Supp. Fig. 3).
• Figure 2: Same vs. other author comparisons, by model.A. Each curve denotes, as a function of the number of training epochs, the the $t$-statistic from a $t$-test comparing the distribution of losses (across random seeds) assigned to held-out texts from the given author (color) versus held-out texts from all other authors. B. The average $t$-statistic across all eight authors, as a function of the number of training epochs. The black curves in both panels indicates the average $t$-value corresponding to $p = 0.001$, for each epoch. Error ribbons denote bootstrap-estimated 95% confidence intervals across authors. See Supplementary Materials for analogous plots using models trained on only content words (Supp. Fig. 4), only function words (Supp. Fig. 5), and only parts of speech (Supp. Fig. 6).
• Figure 3: Confusion matrix. The matrix displays the average cross-entropy loss assigned by models trained on each author's writing (column) to held-out texts from each author (row), after subtracting the native author's baseline loss. See Supplementary Materials for analogous plots using models trained on only content words, function words, and parts of speech (Supp. Fig. 7).
• Figure 4: Multidimensional scaling plot. Three-dimensional MDS projection of the (symmetrized) average cross-entropy loss matrix shown in Figure \ref{['fig:confusion-matrix']}. See Supplementary Materials for analogous plots using models trained on only content words, function words, and parts of speech (Supp. Fig. 8).
• Figure 5: Cross-entropy loss across models and Oz authors. The top sub-panels replicate the Baum (blue) and Thompson (orange) results from Figure \ref{['fig:all-losses']}---i.e., that a given Thompson is well-distinguished from Baum and vice-versa (the two rightmost top sub-panels; error ribbons denote bootstrap-estimated 95% confidence intervals over 10 random seeds). The bottom sub-panels show the cross-entropy loss assigned to a held-out text whose authorship is contested (lower left), to a held-out non-Oz text by Baum (lower center), and to a held-out non-Oz text by Thompson (lower right). I.e., the contested book shows lower loss for Thompson-trained models; a non-Oz Baum book shows lower loss for Baum-trained models; and a non-Oz Thompson book shows lower loss for Thompson-trained models.