Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Writer-R1: Enhancing Generative Writing in LLMs via Memory-augmented Replay Policy Optimization

Jihao Zhao, Shuaishuai Zu, Zhiyuan Ji, Chunlai Zhou, Biao Qin

Abstract

As a typical open-ended generation task, creative writing lacks verifiable reference answers, which has long constrained reward modeling and automatic evaluation due to high human annotation costs, evaluative bias, and coarse feedback signals. To address these challenges, this paper first designs a multi-agent collaborative workflow based on Grounded Theory, performing dimensional decomposition and hierarchical induction of the problem to dynamically produce interpretable and reusable fine-grained criteria. Furthermore, we propose the Memory-augmented Replay Policy Optimization (MRPO) algorithm: on the one hand, without additional training, MRPO guides models to engage in self-reflection based on dynamic criteria, enabling controlled iterative improvement; on the other hand, we adopt the training paradigm that combines supervised fine-tuning with reinforcement learning to convert evaluation criteria into reward signals, achieving end-to-end optimization. Experimental results demonstrate that the automatically constructed criteria achieve performance gains comparable to human annotations. Writer-R1-4B models trained with this approach outperform baselines across multiple creative writing tasks and surpass some 100B+ parameter open-source models.

Writer-R1: Enhancing Generative Writing in LLMs via Memory-augmented Replay Policy Optimization

Abstract

As a typical open-ended generation task, creative writing lacks verifiable reference answers, which has long constrained reward modeling and automatic evaluation due to high human annotation costs, evaluative bias, and coarse feedback signals. To address these challenges, this paper first designs a multi-agent collaborative workflow based on Grounded Theory, performing dimensional decomposition and hierarchical induction of the problem to dynamically produce interpretable and reusable fine-grained criteria. Furthermore, we propose the Memory-augmented Replay Policy Optimization (MRPO) algorithm: on the one hand, without additional training, MRPO guides models to engage in self-reflection based on dynamic criteria, enabling controlled iterative improvement; on the other hand, we adopt the training paradigm that combines supervised fine-tuning with reinforcement learning to convert evaluation criteria into reward signals, achieving end-to-end optimization. Experimental results demonstrate that the automatically constructed criteria achieve performance gains comparable to human annotations. Writer-R1-4B models trained with this approach outperform baselines across multiple creative writing tasks and surpass some 100B+ parameter open-source models.
Paper Structure (29 sections, 19 equations, 3 figures, 6 tables, 1 algorithm)

This paper contains 29 sections, 19 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Writing-Enhanced LLMs
  4. Reward-Driven Optimization
  5. Method
  6. Overview Architecture
  7. Dynamic Criteria Generation
  8. Problem Definition
  9. Aligning Multi-Agent Roles with Grounded-Theory Stages
  10. Open-Coding Agent: Exhaustive Enumeration and Conceptualization of Dimensions
  11. Axial-Coding Agent: Selecting and Merging Key Dimensions
  12. Selective-Coding Agent: Generating Stepwise Rubrics and Actionable Descriptions
  13. MRPO for Self-Reflection
  14. Self-Reflection as Policy Improvement in Text Space
  15. Memory Augmentation: Replacing Parameter Updates with Structured Textual Experience
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Overview of the entire process of our Writer-R1 framework.
  • Figure 2: Comparative analysis of self-reflection strategies across multiple backbone models. The radar plots illustrate the multidimensional performance coverage of MRPO compared to Direct Answering and LLM-as-a-Judge, while the bar chart depicts the distribution of iteration counts required for MRPO.
  • Figure 3: Analysis of two key hyperparameters in the self-reflection and iterative revision process.