A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models

TL;DR

The paper introduces a multi-dimensional constraint framework that captures diverse instruction constraints across three patterns, four categories, and four difficulty levels, paired with an automated pipeline that expands, reconciles, and rewrites instructions to yield 1,200 code-verifiable test cases. It demonstrates substantial variation in instruction-following performance across 19 LLMs and shows that reinforcement learning with constraint-focused data (GRPO) can substantially boost adherence while preserving general capabilities. The authors provide detailed analyses, including parameter-level shifts in attention modules and case studies illustrating how constraint-related information becomes more salient post-training. Collectively, the work offers a scalable path to robust instruction following and interpretable improvements, with code and data available for replication and extension.

Abstract

Instruction following evaluates large language models (LLMs) on their ability to generate outputs that adhere to user-defined constraints. However, existing benchmarks often rely on templated constraint prompts, which lack the diversity of real-world usage and limit fine-grained performance assessment. To fill this gap, we propose a multi-dimensional constraint framework encompassing three constraint patterns, four constraint categories, and four difficulty levels. Building on this framework, we develop an automated instruction generation pipeline that performs constraint expansion, conflict detection, and instruction rewriting, yielding 1,200 code-verifiable instruction-following test samples. We evaluate 19 LLMs across seven model families and uncover substantial variation in performance across constraint forms. For instance, average performance drops from 77.67% at Level I to 32.96% at Level IV. Furthermore, we demonstrate the utility of our approach by using it to generate data for reinforcement learning, achieving substantial gains in instruction following without degrading general performance. In-depth analysis indicates that these gains stem primarily from modifications in the model's attention modules parameters, which enhance constraint recognition and adherence. Code and data are available in https://github.com/Junjie-Ye/MulDimIF.

Figures (4)

• Figure 1: The hierarchical structure of the multi-dimensional constraint framework, which includes three constraint patterns, four constraint categories (subdivided into thirteen subcategories), and four levels of constraint difficulty.
• Figure 2: Illustration of the automated instruction generation pipeline. Constraint Expansion: Randomly selects a constraint category not yet included in the instruction and adds 1–2 specific constraints. Conflict Detection: Identifies whether the new instruction introduces redundant constraints or conflicts, and discards conflicting instructions. Instruction Rewriting: Rewrites the remaining instructions based on different constraint patterns.
• Figure 3: Performance comparison of each LLM on the test sets before and after applying GRPO.
• Figure 4: Parameter change rates of LLMs after GRPO relative to the original ones across different modules.