Modifying Large Language Model Post-Training for Diverse Creative Writing

TL;DR

This work targets the diversity–quality trade-off in creative-writing generation with large language models by introducing deviation-weighted post-training variants of Direct Preference Optimization (DPO) and Odds Ratio Preference Optimization (ORPO). The authors define quality and diversity metrics, and incorporate a per-instance deviation signal to create Diversified DPO (DDPO) and Diversified ORPO (DORPO), encouraging learning from rare high-quality outputs. On the r/writingPrompts dataset, DDPO and DORPO achieve higher output diversity while maintaining quality close to state-of-the-art instruction-tuned models, with human evaluation validating the gains. The approach is robust to data size variations and outperforms a diversification baseline (DivPO) in multiple settings, offering a practical path to diverse, high-quality creative writing from LLMs.

Abstract

As creative writing tasks do not have singular correct answers, large language models (LLMs) trained to perform these tasks should be able to generate diverse valid outputs. However, LLM post-training often focuses on improving generation quality but neglects to facilitate output diversity. Hence, in creative writing generation, we investigate post-training approaches to promote both output diversity and quality. Our core idea is to include deviation -- the degree of difference between a training sample and all other samples with the same prompt -- in the training objective to facilitate learning from rare high-quality instances. By adopting our approach to direct preference optimization (DPO) and odds ratio preference optimization (ORPO), we demonstrate that we can promote the output diversity of trained models while minimally decreasing quality. Our best model with 8B parameters could achieve on-par diversity as a human-created dataset while having output quality similar to the best instruction-tuned models we examined, GPT-4o and DeepSeek-R1. We further validate our approaches with a human evaluation, an ablation, and a comparison to an existing diversification approach, DivPO.

Figures (6)

• Figure 1: Our post-training approach to diversify creative writing generation while maintaining quality.
• Figure 2: Results on writing quality (reddit-reward, $x$ axes) and diversity (semantic or style diversity, $y$ axes). Error bars in this paper indicate 95% confidence intervals.
• Figure 3: Ablation results by varying the maximum number of responses per prompt. When the maximum number of responses is four, we also experimented with 1) setting a minimum $\delta$ and 2) using high-quality responses.
• Figure 4: Llama-3.1-8B results on compression ratio, homogenization score, and n-gram diversity score.
• Figure 5: Mistral-7B-v0.3 results on compression ratio, homogenization score, and n-gram diversity score.