Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating LLM Story Generation through Large-scale Network Analysis of Social Structures

Hiroshi Nonaka, K. E. Perry

TL;DR

Problem: scalable evaluation of LLM storytelling is difficult with human judgments. Approach: build signed character networks from narrative co-occurrences in over 1,200 stories (LLMs and humans), and analyze density, clustering, and edge polarity, including positive/negative subgraphs and Wasserstein distances. Key findings: LLM-generated stories exhibit higher density and clustering and a positivity bias relative to human-written stories, consistent with prior human assessments. Significance: the network-based analysis provides a scalable, quantitative tool for assessing LLM creativity and can be extended to other genres, longer texts, and interactive narratives.

Abstract

Evaluating the creative capabilities of large language models (LLMs) in complex tasks often requires human assessments that are difficult to scale. We introduce a novel, scalable methodology for evaluating LLM story generation by analyzing underlying social structures in narratives as signed character networks. To demonstrate its effectiveness, we conduct a large-scale comparative analysis using networks from over 1,200 stories, generated by four leading LLMs (GPT-4o, GPT-4o mini, Gemini 1.5 Pro, and Gemini 1.5 Flash) and a human-written corpus. Our findings, based on network properties like density, clustering, and signed edge weights, show that LLM-generated stories consistently exhibit a strong bias toward tightly-knit, positive relationships, which aligns with findings from prior research using human assessment. Our proposed approach provides a valuable tool for evaluating limitations and tendencies in the creative storytelling of current and future LLMs.

Evaluating LLM Story Generation through Large-scale Network Analysis of Social Structures

TL;DR

Problem: scalable evaluation of LLM storytelling is difficult with human judgments. Approach: build signed character networks from narrative co-occurrences in over 1,200 stories (LLMs and humans), and analyze density, clustering, and edge polarity, including positive/negative subgraphs and Wasserstein distances. Key findings: LLM-generated stories exhibit higher density and clustering and a positivity bias relative to human-written stories, consistent with prior human assessments. Significance: the network-based analysis provides a scalable, quantitative tool for assessing LLM creativity and can be extended to other genres, longer texts, and interactive narratives.

Abstract

Evaluating the creative capabilities of large language models (LLMs) in complex tasks often requires human assessments that are difficult to scale. We introduce a novel, scalable methodology for evaluating LLM story generation by analyzing underlying social structures in narratives as signed character networks. To demonstrate its effectiveness, we conduct a large-scale comparative analysis using networks from over 1,200 stories, generated by four leading LLMs (GPT-4o, GPT-4o mini, Gemini 1.5 Pro, and Gemini 1.5 Flash) and a human-written corpus. Our findings, based on network properties like density, clustering, and signed edge weights, show that LLM-generated stories consistently exhibit a strong bias toward tightly-knit, positive relationships, which aligns with findings from prior research using human assessment. Our proposed approach provides a valuable tool for evaluating limitations and tendencies in the creative storytelling of current and future LLMs.
Paper Structure (28 sections, 5 equations, 2 figures, 2 tables, 1 algorithm)

This paper contains 28 sections, 5 equations, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. LLMs in creative writing
  4. Character network analysis
  5. Methodology
  6. Short story generation
  7. LLM short stories
  8. Human short stories
  9. Network extraction
  10. Graph structure
  11. Vertex contractions
  12. Exclusion criteria
  13. Network analysis
  14. Results
  15. Distribution analysis
  16. ...and 13 more sections

Figures (2)

  • Figure 1: Violin plots of connectivity measure distributions. The horizontal axes show writers: from the left, Gemini 1.5 Flash (green), GPT 4o mini (yellow), Gemini 1.5 Pro (purple), GPT 4o (red), and Project Gutenberg (blue). LLM score distributions cluster in the somewhat same range.
  • Figure 2: Wasserstein distances (WD) between pairs of distributions for the connectivity measures. Overall, human-written stories have the highest distances with LLMs, while the models have relatively close distributions with each other.