LSRIF: Logic-Structured Reinforcement Learning for Instruction Following

TL;DR

LSRIF tackles instruction following with complex logical structures by introducing a logic-structured data pipeline (LsrInstruct) and a structure-aware reward model (LsRM). It defines parallel, sequential, and conditional constraint forms and tailors reward signals accordingly, using GRPO to optimize policy updates. Across model scales and diverse benchmarks, LSRIF improves both instruction following and general reasoning, with ablations highlighting the critical role of structure-aware rewards and dataset construction. Interpretability analyses show training biases attention toward logical connectors and constraint tokens, suggesting explicit logic modeling strengthens both performance and explainability.

Abstract

Instruction-following is critical for large language models, but real-world instructions often contain logical structures such as sequential dependencies and conditional branching. Existing methods typically construct datasets with parallel constraints and optimize average rewards, ignoring logical dependencies and yielding noisy signals. We propose a logic-structured training framework LSRIF that explicitly models instruction logic. We first construct a dataset LSRInstruct with constraint structures such as parallel, sequential, and conditional types, and then design structure-aware rewarding method LSRIF including average aggregation for parallel structures, failure-penalty propagation for sequential structures, and selective rewards for conditional branches. Experiments show LSRIF brings significant improvements in instruction-following (in-domain and out-of-domain) and general reasoning. Analysis reveals that learning with explicit logic structures brings parameter updates in attention layers and sharpens token-level attention to constraints and logical operators.

Figures (9)

• Figure 1: Essentially, the complex instruction is the logical composition of constraints.
• Figure 2: Our framework LsrIF consists of two components: (LsrInstruct) logic-structured dataset construction, and (LsRM) structure-aware reward modeling with corresponding methods.
• Figure 3: LsrIF performance on different reward forms. Const.-Level and Inst.-Level refer to constraint-level and instruction-level, respectively.
• Figure 4: Performance on nested structures from wen2024benchmarking.
• Figure 5: Parameter change rates of LLMs to the original ones across different modules. Darker orange colors indicate larger parameter changes.