Rewarding Creativity: A Human-Aligned Generative Reward Model for Reinforcement Learning in Storytelling

TL;DR

This work tackles the core challenges of applying reinforcement learning to creative storytelling by separating reward design from policy optimization. It introduces Generative Reward Model (GenRM), a two-stage system that provides multi-dimensional, reasoned evaluations of story quality via supervised fine-tuning and group-relative policy optimization, achieving up to $68\%$ alignment with human judgments. To stabilize learning in subjective tasks, it adds an entropy-based reward shaping scheme that focuses on informative errors and uncertain corrects while preventing overfitting. Experiments show RLCS markedly improves story quality over strong baselines, including Gemini-2.5-Pro, demonstrating a practical, scalable pipeline for RL in open-ended language generation tasks. This framework offers a generalizable approach to reward modeling and stable RL in creative domains, with potential for broader applicability beyond storytelling.

Abstract

While Large Language Models (LLMs) can generate fluent text, producing high-quality creative stories remains challenging. Reinforcement Learning (RL) offers a promising solution but faces two critical obstacles: designing reliable reward signals for subjective storytelling quality and mitigating training instability. This paper introduces the Reinforcement Learning for Creative Storytelling (RLCS) framework to systematically address both challenges. First, we develop a Generative Reward Model (GenRM) that provides multi-dimensional analysis and explicit reasoning about story preferences, trained through supervised fine-tuning on demonstrations with reasoning chains distilled from strong teacher models, followed by GRPO-based refinement on expanded preference data. Second, we introduce an entropy-based reward shaping strategy that dynamically prioritizes learning on confident errors and uncertain correct predictions, preventing overfitting on already-mastered patterns. Experiments demonstrate that GenRM achieves 68\% alignment with human creativity judgments, and RLCS significantly outperforms strong baselines including Gemini-2.5-Pro in overall story quality. This work provides a practical pipeline for applying RL to creative domains, effectively navigating the dual challenges of reward modeling and training stability.

Figures (5)

• Figure 1: Overview of the RLCS Framework: (a) Data Construction from unlabeled stories and human feedback, (b) Generative Reward Model (GenRM) training with rule-based rewards, and (c) Story Model training via supervised fine-tuning and reinforcement learning with GenRM guidance.
• Figure 2: Impact of group rollout size on GenRM performance. Performance saturates around $G=8$.
• Figure 3: Training dynamics of GenRM across different model scales. The GRPO phase consistently improves upon SFT initialization.
• Figure 4: Prompt template for automated story preference labeling.
• Figure 5: Prompt template for automated story preference labeling translated into English.