Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models

TL;DR

The paper addresses the challenge of following complex instructions by large language models, where vanilla chain-of-thought can degrade performance. It introduces RAIF, a systematic framework that incentivizes deep reasoning through reinforcement learning with verifiable rewards and an instruction-evolving pipeline that generates diverse, rule-checked prompts. By combining rule-centric rewards, superior CoT enforcement, experience replay filtering, and behavior cloning, RAIF demonstrates robust gains across seven benchmarks with a 1.5B model achieving substantial improvements and generalization to out-of-distribution constraints. The work offers practical guidance for constructing scalable instruction data and RL methodologies that promote genuine reasoning over superficial paraphrasing, with broad implications for advanced instruction-following in heterogeneous model families.

Abstract

Existing large language models (LLMs) face challenges of following complex instructions, especially when multiple constraints are present and organized in paralleling, chaining, and branching structures. One intuitive solution, namely chain-of-thought (CoT), is expected to universally improve capabilities of LLMs. However, we find that the vanilla CoT exerts a negative impact on performance due to its superficial reasoning pattern of simply paraphrasing the instructions. It fails to peel back the compositions of constraints for identifying their relationship across hierarchies of types and dimensions. To this end, we propose RAIF, a systematic method to boost LLMs in dealing with complex instructions via incentivizing reasoning for test-time compute scaling. First, we stem from the decomposition of complex instructions under existing taxonomies and propose a reproducible data acquisition method. Second, we exploit reinforcement learning (RL) with verifiable rule-centric reward signals to cultivate reasoning specifically for instruction following. We address the shallow, non-essential nature of reasoning under complex instructions via sample-wise contrast for superior CoT enforcement. We also exploit behavior cloning of experts to facilitate steady distribution shift from fast-thinking LLMs to skillful reasoners. Extensive evaluations on seven comprehensive benchmarks confirm the validity of the proposed method, where a 1.5B LLM achieves 11.74% gains with performance comparable to a 8B LLM. Evaluation on OOD constraints also confirms the generalizability of our RAIF. Codes and data are available at https://github.com/yuleiqin/RAIF. Keywords: reinforcement learning with verifiable rewards (RLVR), instruction following, complex instructions

Figures (36)

• Figure 1: Complex instructions with various atomic constraints and compositions pose great challenges to instruction-following capabilities of LLMs (The above example and its structure are from the ComplexBench wen2024benchmarking). Our preliminary experiments demonstrate that the CoT prompting of existing LLMs often elicits shallow reasoning that blindly, mechanically responds to the request without formulation of structured analyses. In contrast to R1 and QwQ, most fask-thinking models cannot benefit from the vanilla CoT at all due to such superficial nature (see Sec. \ref{['sec:empirical']}). Our proposed method boosts deep reasoning of both fast- and slow-thinking LLMs under complex instructions.
• Figure 2: Illustration of the proposed method for advanced instruction-following via reasoning.
• Figure 3: The averaged number of reasoning tokens and scores over steps (best viewed magnified).
• Figure 4: The averaged frequency change of keyword tokens of DeepSeek-Qwen1.5B, DeepScaleR-1.5B, Qwen2.5-1.5B-Instruct, and Qwen2.5-7B-Instruct before/after RL (best viewed magnified).
• Figure 5: The ratio of samples kept by superior CoT and the total reward over steps of Qwen2.5-7B-Instruct (best viewed magnified).