Tailored Primitive Initialization is the Secret Key to Reinforcement Learning

TL;DR

The paper tackles the dependence of reinforcement learning for large language models on initialization and sampling efficiency. It introduces Tailor, a finetuning pipeline that automatically synthesizes diverse, high-quality reasoning primitives to expand thinking-trajectory coverage before RL, operationalized via a KL-regularized clipped objective ${\mathcal{J}}_{\text{RL}}$. Through experiments on KK and iGSM benchmarks with multiple backbones, Tailor yields warmer-start data that leads to stronger downstream RL performance and faster convergence than rule-based, 4-STaR, or re-distillation baselines. The work emphasizes data-centric RL and demonstrates that improving the diversity and quality of reasoning primitives at SFT time can substantially boost RL efficiency and outcomes, while acknowledging domain limitations and safety considerations for future research.

Abstract

Reinforcement learning (RL) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs). While RL has demonstrated substantial performance gains, it still faces key challenges, including low sampling efficiency and a strong dependence on model initialization: some models achieve rapid improvements with minimal RL steps, while others require significant training data to make progress. In this work, we investigate these challenges through the lens of reasoning token coverage and argue that initializing LLMs with diverse, high-quality reasoning primitives is essential for achieving stable and sample-efficient RL training. We propose Tailor, a finetuning pipeline that automatically discovers and curates novel reasoning primitives, thereby expanding the coverage of reasoning-state distributions before RL. Extensive experiments on mathematical and logical reasoning benchmarks demonstrate that Tailor generates more diverse and higher-quality warm-start data, resulting in higher downstream RL performance.

Figures (10)

• Figure 1: Coverage comparison. The goal in the maze refers to the correct answer, and the trajectories refer to the thinking tokens.
• Figure 2: Overview of the Tailor finetuning pipeline.
• Figure 3: Examples of reasoning primitives.
• Figure 4: Training Curves of the KK tasks. We average curves with $3$ random seeds.
• Figure 5: The evaluation results (%) on the KK reasoning tasks. We train SFT models for $4$ epochs and finetune them with RL for 5 epochs. The mean value and standard deviation are calculated over $3$ random seeds.