Tell, Don't Show: Leveraging Language Models' Abstractive Retellings to Model Literary Themes

TL;DR

The paper addresses the challenge of topic modeling literary texts by leveraging abstractive retellings produced by resource-efficient language models. By prompting LMs to describe, summarize, or paraphrase passages and then applying LDA to these retellings, the authors surface higher-level themes that better reflect literary content than traditional LDA or direct LM topic labeling. Across a large, multi-source corpus and a focused case study on racial/cultural identity in English language arts texts, Retell demonstrates higher topic-relatedness and interpretive usefulness, while remaining accessible to humanities researchers with limited computational resources. The work includes comprehensive evaluations, a publicly released codebase, and a discussion of limitations and future directions for scalable literary content analysis.

Abstract

Conventional bag-of-words approaches for topic modeling, like latent Dirichlet allocation (LDA), struggle with literary text. Literature challenges lexical methods because narrative language focuses on immersive sensory details instead of abstractive description or exposition: writers are advised to "show, don't tell." We propose Retell, a simple, accessible topic modeling approach for literature. Here, we prompt resource-efficient, generative language models (LMs) to tell what passages show, thereby translating narratives' surface forms into higher-level concepts and themes. By running LDA on LMs' retellings of passages, we can obtain more precise and informative topics than by running LDA alone or by directly asking LMs to list topics. To investigate the potential of our method for cultural analytics, we compare our method's outputs to expert-guided annotations in a case study on racial/cultural identity in high school English language arts books.

Figures (7)

• Figure 1: Language models (LMs) can support traditional topic modeling pipelines, e.g. LDA, by translating low-level sensory detail (showing) into high-level abstractive exposition (telling). This example book excerpt is from Little Fires Everywhere by Celeste Ng.
• Figure 2: Precision and recall ($y$-axes) of using relevant Retell and LDA topics' probability thresholds ($x$-axes) to distinguish different types of human-labeled passages from each other (discuss, mention, neither). Examining against compares only one topic per method, and is more precise. Comparing with considers all relevant topics each method surfaced, and has higher recall without sacrificing precision. Left two plots: identifying passages that discuss race. Right two plots: identifying passages that either discuss or mention race.
• Figure 3: Additional evaluation results for $k=50$, 100, and 200, disaggregated by the source of ground truth labels. Dashed gray lines indicate default LDA's performance, and the models on the $x$-axis include Llama 3.1 8B (L), Phi-3.5-mini (P), Gemma 2 2B (G) and GPT-4o mini (O). Since TopicGPT-lite is not designed for use with predetermined $k$, it is not included in these plots.
• Figure 4: Results for two different runs of Llama 3.1 8B (L) and GPT-4o mini (O). Dashed gray lines indicate default LDA's performance.
• Figure 5: Prompt for generating a possible topic pool over 1k passages. We truncate the few-shot examples here for brevity (with "..."), but show the beginnings and ends to support their reconstruction. The first example is from Martin Luther King Jr.'s Stride Toward Freedom, while the second is from Jane Austen's Emma.