Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Retell, Reward, Repeat: Reinforcement Learning for Narrative Theory-Informed Story Generation

David Y. Liu, Xanthe Muston, Aditya Joshi, Sebastian Sequoiah-Grayson

TL;DR

This work addresses the subjectivity of automatic story generation by grounding post-training in narrative theory, specifically Todorov's Narrative Equilibrium, and evaluating with a theory-informed LLM-as-judge. It introduces d-RLAIF as an alternative to supervised fine-tuning, training 7–8B models using a GRPO-LoRA pipeline and narrativity-based rewards on the TimeTravel dataset. Key contributions include a formal annotation scheme, a human-vs-LLM evaluation study (n=200), and comprehensive experiments showing d-RLAIF can yield more diverse, narratively aligned outputs than SFT, with RN (narrativity) rewards performing best on several metrics. The findings suggest that theory-guided reinforcement learning is a promising direction for subjective tasks like ASG and can reduce data requirements while fostering linguistically grounded post-training and better alignment with human narrative conventions.

Abstract

Despite the subjective nature of storytelling, past works on automatic story generation (ASG) have relied on limited ground truths for training and evaluation. In this work, we explore reinforcement learning (d-RLAIF) as a post-training alternative to supervised fine-tuning (SFT). We first apply Todorov's Theory of Narrative Equilibrium to establish principles that define desirable ASG qualities. We prompt 7B and 14B LLM-as-judge models with our principles to test alignment with human annotators and provide reward signals during d-RLAIF. We use Gemini-3-Flash to evaluate the output of our post-trained models and compare them to human-written stories from the TimeTravel dataset. We show that d-RLAIF offers a viable alternative to supervised fine-tuning (SFT)--producing stories that are more diverse and aligned with human narrative conventions. Our paper demonstrates the promise of reinforcement learning for linguistically grounded post-training for subjective tasks such as ASG.

Retell, Reward, Repeat: Reinforcement Learning for Narrative Theory-Informed Story Generation

TL;DR

This work addresses the subjectivity of automatic story generation by grounding post-training in narrative theory, specifically Todorov's Narrative Equilibrium, and evaluating with a theory-informed LLM-as-judge. It introduces d-RLAIF as an alternative to supervised fine-tuning, training 7–8B models using a GRPO-LoRA pipeline and narrativity-based rewards on the TimeTravel dataset. Key contributions include a formal annotation scheme, a human-vs-LLM evaluation study (n=200), and comprehensive experiments showing d-RLAIF can yield more diverse, narratively aligned outputs than SFT, with RN (narrativity) rewards performing best on several metrics. The findings suggest that theory-guided reinforcement learning is a promising direction for subjective tasks like ASG and can reduce data requirements while fostering linguistically grounded post-training and better alignment with human narrative conventions.

Abstract

Despite the subjective nature of storytelling, past works on automatic story generation (ASG) have relied on limited ground truths for training and evaluation. In this work, we explore reinforcement learning (d-RLAIF) as a post-training alternative to supervised fine-tuning (SFT). We first apply Todorov's Theory of Narrative Equilibrium to establish principles that define desirable ASG qualities. We prompt 7B and 14B LLM-as-judge models with our principles to test alignment with human annotators and provide reward signals during d-RLAIF. We use Gemini-3-Flash to evaluate the output of our post-trained models and compare them to human-written stories from the TimeTravel dataset. We show that d-RLAIF offers a viable alternative to supervised fine-tuning (SFT)--producing stories that are more diverse and aligned with human narrative conventions. Our paper demonstrates the promise of reinforcement learning for linguistically grounded post-training for subjective tasks such as ASG.
Paper Structure (21 sections, 1 equation, 5 figures, 4 tables)

This paper contains 21 sections, 1 equation, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Todorov's Theory of Narrative Equilibrium
  4. Data Annotation
  5. Criteria
  6. Curation
  7. Human Annotation
  8. LLM-as-judge
  9. Harnessing d-RLAIF for ASG
  10. Results
  11. Discussion
  12. Related Work
  13. Conclusion and Future Work
  14. License
  15. TimeTravel Dataset
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Our architecture that utilises d-RLAIF lee2024rlaif w/ a narrative theory-informed LLM-as-judge generating the reward signal for GRPO shao2024deepseekmath
  • Figure 2: Mean and standard deviation of the reward signal RO generated by Selene-1-mini during d-RLAIF.
  • Figure 6: LLM-as-judge on the annotation set. Left: 'Narrativity' distribution. Right: 'Overall' distribution.
  • Figure 7: Correlation between each criteria in the human annotations.
  • Figure 8: Policy model gradient norm during d-RLAIF using the reward signal RO generated by Selene-1-mini.