Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning

Yuhao Wu, Yushi Bai, Zhiqiang Hu, Roy Ka-Wei Lee, Juanzi Li

TL;DR

This work proposes an incentivization-based approach that leverages reinforcement learning (RL) to foster the emergence of ultra-long, high-quality text generation capabilities in LLMs, and employs specialized reward models that steer the LLM towards improved length control, writing quality, and structural formatting.

Abstract

Ultra-long generation by large language models (LLMs) is a widely demanded scenario, yet it remains a significant challenge due to their maximum generation length limit and overall quality degradation as sequence length increases. Previous approaches, exemplified by LongWriter, typically rely on ''teaching'', which involves supervised fine-tuning (SFT) on synthetic long-form outputs. However, this strategy heavily depends on synthetic SFT data, which is difficult and costly to construct, often lacks coherence and consistency, and tends to be overly artificial and structurally monotonous. In this work, we propose an incentivization-based approach that, starting entirely from scratch and without relying on any annotated or synthetic data, leverages reinforcement learning (RL) to foster the emergence of ultra-long, high-quality text generation capabilities in LLMs. We perform RL training starting from a base model, similar to R1-Zero, guiding it to engage in reasoning that facilitates planning and refinement during the writing process. To support this, we employ specialized reward models that steer the LLM towards improved length control, writing quality, and structural formatting. Experimental evaluations show that our LongWriter-Zero model, trained from Qwen2.5-32B, consistently outperforms traditional SFT methods on long-form writing tasks, achieving state-of-the-art results across all metrics on WritingBench and Arena-Write, and even surpassing 100B+ models such as DeepSeek R1 and Qwen3-235B. We open-source our data and model checkpoints under https://huggingface.co/THU-KEG/LongWriter-Zero-32B

LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning

TL;DR

This work proposes an incentivization-based approach that leverages reinforcement learning (RL) to foster the emergence of ultra-long, high-quality text generation capabilities in LLMs, and employs specialized reward models that steer the LLM towards improved length control, writing quality, and structural formatting.

Abstract

Ultra-long generation by large language models (LLMs) is a widely demanded scenario, yet it remains a significant challenge due to their maximum generation length limit and overall quality degradation as sequence length increases. Previous approaches, exemplified by LongWriter, typically rely on ''teaching'', which involves supervised fine-tuning (SFT) on synthetic long-form outputs. However, this strategy heavily depends on synthetic SFT data, which is difficult and costly to construct, often lacks coherence and consistency, and tends to be overly artificial and structurally monotonous. In this work, we propose an incentivization-based approach that, starting entirely from scratch and without relying on any annotated or synthetic data, leverages reinforcement learning (RL) to foster the emergence of ultra-long, high-quality text generation capabilities in LLMs. We perform RL training starting from a base model, similar to R1-Zero, guiding it to engage in reasoning that facilitates planning and refinement during the writing process. To support this, we employ specialized reward models that steer the LLM towards improved length control, writing quality, and structural formatting. Experimental evaluations show that our LongWriter-Zero model, trained from Qwen2.5-32B, consistently outperforms traditional SFT methods on long-form writing tasks, achieving state-of-the-art results across all metrics on WritingBench and Arena-Write, and even surpassing 100B+ models such as DeepSeek R1 and Qwen3-235B. We open-source our data and model checkpoints under https://huggingface.co/THU-KEG/LongWriter-Zero-32B

Paper Structure

This paper contains 42 sections, 5 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Reinforcement Learning for Ultra-Long Text Generation
  3. RL Setup
  4. Training Algorithm.
  5. Query Source.
  6. Training Configuration.
  7. Evaluation (Arena-Write).
  8. RQ1: Reward Design
  9. Length RM.
  10. Writing RM.
  11. Format RM.
  12. Final Reward.
  13. Result.
  14. RQ2: Test-time Scaling
  15. Result.
  16. ...and 27 more sections

Figures (6)

  • Figure 1: SFT vs. RL in long-form generation.
  • Figure 2: RL Training curves of three setups (Base-nothink, Base-think, and Continual-Pretrain-think) across three metrics: Writing RM (left), Length RM (middle), and Mean Non-Overlong Generation Length (right).
  • Figure 3: Elo scores evaluated on Arena-Write during training for the three setups: Base-nothink, Base-think, and Continual-Pretrain-think. The y-axis shows the Elo score, and the x-axis represents training steps.
  • Figure 4: Arena-Write performance across RL training steps, comparing RL (solid) and SFT (dashed) starting from Base (orange) and Continual Pretrain (blue) initializations.
  • Figure 5: Win-rate results of LongWriter-Zero in human-in-the-loop win-rate evaluation. Left six charts: Outcomes judged by GPT-4.1 against six baselines (Llama-4-Scout, DeepSeek-V3, DeepSeek-R1, Claude-Sonnet-4, Gemini-2.5-Pro, Qwen3-235B-A22B). Right two charts: Outcomes judged by human annotators (comparing against DeepSeek-R1 and Qwen3-235B-A22B). The percentage in the center indicates the overall win rate, with ties counted as 0.5 wins for each side.
  • ...and 1 more figures