DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writing

TL;DR

DPWriter tackles diversity collapse in RL-enhanced LLMs for open-ended creative writing by introducing a semi-structured long Chain-of-Thought with an explicit planning stage and a Diverse Planning Branching mechanism. The approach uses a GRPO-based RL objective plus a planning-aware diversity reward to encourage exploration of diverse, high-quality trajectories. Empirical results on multiple creative-writing benchmarks show significant gains in diversity (both lexical and semantic) without sacrificing output quality, outperforming strong baselines across backbones. This framework offers a principled, controllable method for enhancing expressiveness in open-ended generation with potential applications beyond creative writing.

Abstract

Reinforcement learning (RL)-based enhancement of large language models (LLMs) often leads to reduced output diversity, undermining their utility in open-ended tasks like creative writing. Current methods lack explicit mechanisms for guiding diverse exploration and instead prioritize optimization efficiency and performance over diversity. This paper proposes an RL framework structured around a semi-structured long Chain-of-Thought (CoT), in which the generation process is decomposed into explicitly planned intermediate steps. We introduce a Diverse Planning Branching method that strategically introduces divergence at the planning phase based on diversity variation, alongside a group-aware diversity reward to encourage distinct trajectories. Experimental results on creative writing benchmarks demonstrate that our approach significantly improves output diversity without compromising generation quality, consistently outperforming existing baselines.

Figures (16)

• Figure 1: Comparison among three generation paradigms. Our semi-structured reasoning paradigm introduces global planning before reasoning, providing high-level guidance while maintaining higher quality.
• Figure 2: An example of the original long CoT data and the semi-structured long CoT with planning. Texts with HTML]05FE04colored background represent special tokens.
• Figure 3: An overview of our Diverse Planning Branching method. During RL, at each planning segment, we branch out multiple diverse continuations from each candidate, forming a pool of candidates. We then select the most diverse ones to proceed to the next segment, ultimately generating diverse final responses.
• Figure 4: NoveltyBench results comparing DPWriter with baselines diversity metric Distinct.
• Figure 5: Case study comparing DPWriter with GRPO on a sample from NoveltyBench. The same answer is highlighted in the same colored background. We present five generations from each model for comparison.