RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Data

TL;DR

This work targets the bottleneck of complex instruction following by LLMs. It introduces RECAST, a scalable pipeline that mines both rule-based and model-based constraints from real responses to produce RECAST-30K, a dense dataset with up to 19 constraint types across roughly 29,939 examples. By enabling automatic constraint verification via rule-based and LLM validators, RECAST supports per-constraint rewards and reinforcement learning. The authors propose RLVC, which uses GRPO to optimize multi-constraint satisfaction while preserving general capabilities, achieving strong performance on RECAST-Test, out-of-domain benchmarks, and general capability evaluations. Overall, RECAST provides verifiable supervision and scalable resources to push the practical reliability and controllability of instruction-following in LLMs.

Abstract

Large language models (LLMs) are increasingly expected to tackle complex tasks, driven by their expanding applications and users' growing proficiency in crafting sophisticated prompts. However, as the number of explicitly stated requirements increases (particularly more than 10 constraints), LLMs often struggle to accurately follow such complex instructions, which limits their applicability in complex real-world scenarios. To the best of our knowledge, existing datasets do not exceed 10 constraints per instance. To address this challenge, we propose RECAST, an efficient and scalable framework for synthesizing datasets where each example incorporates far more constraints than those in existing benchmarks, aiming to challenge and extend the boundaries of models' ability to follow complex instructions. These constraints are extracted from real-world prompt-response pairs to ensure practical relevance. Using this framework, we construct RECAST-30K, a large-scale, high-quality dataset comprising 30k instances spanning 19 constraint types. Experimental results demonstrate that models finetuned on RECAST-30K substantially improve in following complex instructions while maintaining their general capabilities without degradation. Moreover, RECAST enables automatic verification of constraint satisfaction via rule-based validators for quantitative constraints and LLM-based validators for qualitative ones; the verifiability provided by RECAST enables the design of reward functions for reinforcement learning, which further boosts model performance on complex and challenging tasks.

Figures (26)

• Figure 1: Challenges in complex instruction following and improved performance via RECAST-30k. Left: Real-world examples illustrating how LLMs fail to follow complex instructions. Top-right: Performance degradation of LLMs as the number of constraints increases. Bottom-right: Comparison of instruction-following performance between models fine-tuned on RECAST and their corresponding Instruct variants (e.g., Qwen2.5-7B-Instruct).
• Figure 2: Overview of the RECAST framework and RLVC. Left: The RECAST pipeline generates complex instruction-following data through four steps: (1) seed data collection across diverse domains, (2) constraint construction with both rule-based and model-based verification methods, (3) instruction enhancement by integrating selected constraints, and (4) response synthesis ensuring constraints are satisfied. More detailed descriptions of the pipeline are provided in Appendix \ref{['appendix:description_of_RECAST']}. Top-right: Using RECAST-generated data to fine-tune LLMs through SFT. Bottom-right: RLVC framework leveraging constraint-specific verification to provide fine-grained rewards, guiding model optimization toward satisfying multiple constraints simultaneously.
• Figure 3: Overview of the RECAST framework. The RECAST pipeline generates complex instruction-following data through four steps: (1) seed data collection across diverse domains, (2) constraint construction with both rule-based and model-based verification methods, (3) instruction enhancement by integrating selected constraints, and (4) response synthesis ensuring constraints are satisfied.
• Figure 4: Constraint Count Distribution of RECAST-30K.
• Figure 5: Constraint Count Distribution of Train Set with 5 Constraints.