Applying Large Language Models to Characterize Public Narratives

TL;DR

The paper introduces a codebook-guided LLM framework to automate Public Narrative (PN) annotation, validates near-human performance against PN experts on eight narratives across 14 codes, and scales the approach to 22 narratives and two political speeches. It formalizes the PN structure (Story of Self/Us/Now and Challenge–Choice–Outcome) and empirically analyzes code frequencies, correlations, and extrapolations to real-world political rhetoric. Key findings show that LLMs perform well on explicit, frequent codes but struggle with subjective elements, highlighting the importance of a structured rubric and prompting strategy. The work demonstrates the potential of LLM-assisted PN annotation for scalable civic storytelling analysis with implications for leadership research and computational social science.

Abstract

Public Narratives (PNs) are key tools for leadership development and civic mobilization, yet their systematic analysis remains challenging due to their subjective interpretation and the high cost of expert annotation. In this work, we propose a novel computational framework that leverages large language models (LLMs) to automate the qualitative annotation of public narratives. Using a codebook we co-developed with subject-matter experts, we evaluate LLM performance against that of expert annotators. Our work reveals that LLMs can achieve near-human-expert performance, achieving an average F1 score of 0.80 across 8 narratives and 14 codes. We then extend our analysis to empirically explore how PN framework elements manifest across a larger dataset of 22 stories. Lastly, we extrapolate our analysis to a set of political speeches, establishing a novel lens in which to analyze political rhetoric in civic spaces. This study demonstrates the potential of LLM-assisted annotation for scalable narrative analysis and highlights key limitations and directions for future research in computational civic storytelling.

Figures (12)

• Figure 1: We compare an LLMs' ability to annotate Public Narratives to human experts following our codebook co-developed with experts. To visualize the annotations, the length of each portion corresponds to the number of sentences coded.
• Figure 2: The average F1 scores per code of o3-mini with CoT + Prompt Chaining compared to Minimum Match human annotations across the eight annotated stories. The bars represent the standard deviation across 3 runs.
• Figure 3: Comparison of a Donald Trump and Jacinda Ardern political speech automatically annotated using o3-mini with our PN codebook.
• Figure 4: Pearson correlation heatmap of code co-occurrence for o3-mini with CoT + Prompt Chaining, averaged across the 22 PNs.
• Figure 5: Jaccard similarity heatmap of code co-occurrence for o3-mini with CoT + Prompt Chaining, averaged across the 22 PNs.