Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilities

Alberto Purpura, Li Wang, Sahil Badyal, Eugenio Beaufrand, Adam Faulkner

TL;DR

This work introduces MOSAIC, a modular benchmark designed to isolate and diagnose instruction compliance in large language models by decoupling compliance from task performance through dynamically generated, application-oriented constraints. The framework builds a large, synthetic dataset and uses a suite of metrics (SCC, PCC, PosCC, PA) along with an LLM-as-a-judge to assess both single and interacting constraints, including their positions within prompts. Empirical results across multiple LLMs reveal that instruction-following is not monolithic; compliance varies with constraint type, quantity, and position, and models exhibit distinct positional biases (primacy vs. recency) and interaction patterns. MOSAIC thus provides a granular diagnostic tool with practical implications for prompt engineering and the development of more reliable, instruction-compliant LLM systems, while acknowledging limitations related to judge bias and synthetic data. It offers a pathway to targeted improvements in prompt design and model training to ensure strict adherence to complex instruction sets in real-world applications.

Abstract

Reliably ensuring Large Language Models (LLMs) follow complex instructions is a critical challenge, as existing benchmarks often fail to reflect real-world use or isolate compliance from task success. We introduce MOSAIC (MOdular Synthetic Assessment of Instruction Compliance), a modular framework that uses a dynamically generated dataset with up to 20 application-oriented generation constraints to enable a granular and independent analysis of this capability. Our evaluation of five LLMs from different families based on this new benchmark demonstrates that compliance is not a monolithic capability but varies significantly with constraint type, quantity, and position. The analysis reveals model-specific weaknesses, uncovers synergistic and conflicting interactions between instructions, and identifies distinct positional biases such as primacy and recency effects. These granular insights are critical for diagnosing model failures and developing more reliable LLMs for systems that demand strict adherence to complex instructions.

Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilities

TL;DR

This work introduces MOSAIC, a modular benchmark designed to isolate and diagnose instruction compliance in large language models by decoupling compliance from task performance through dynamically generated, application-oriented constraints. The framework builds a large, synthetic dataset and uses a suite of metrics (SCC, PCC, PosCC, PA) along with an LLM-as-a-judge to assess both single and interacting constraints, including their positions within prompts. Empirical results across multiple LLMs reveal that instruction-following is not monolithic; compliance varies with constraint type, quantity, and position, and models exhibit distinct positional biases (primacy vs. recency) and interaction patterns. MOSAIC thus provides a granular diagnostic tool with practical implications for prompt engineering and the development of more reliable, instruction-compliant LLM systems, while acknowledging limitations related to judge bias and synthetic data. It offers a pathway to targeted improvements in prompt design and model training to ensure strict adherence to complex instruction sets in real-world applications.

Abstract

Reliably ensuring Large Language Models (LLMs) follow complex instructions is a critical challenge, as existing benchmarks often fail to reflect real-world use or isolate compliance from task success. We introduce MOSAIC (MOdular Synthetic Assessment of Instruction Compliance), a modular framework that uses a dynamically generated dataset with up to 20 application-oriented generation constraints to enable a granular and independent analysis of this capability. Our evaluation of five LLMs from different families based on this new benchmark demonstrates that compliance is not a monolithic capability but varies significantly with constraint type, quantity, and position. The analysis reveals model-specific weaknesses, uncovers synergistic and conflicting interactions between instructions, and identifies distinct positional biases such as primacy and recency effects. These granular insights are critical for diagnosing model failures and developing more reliable LLMs for systems that demand strict adherence to complex instructions.
Paper Structure (25 sections, 4 equations, 5 figures, 13 tables)

This paper contains 25 sections, 4 equations, 5 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Approach
  4. Dataset Generation
  5. Evaluation Metrics
  6. Evaluation Strategy
  7. Evaluation Results
  8. Single Constraints Compliance Analysis
  9. Pairwise Constraint Compliance Analysis
  10. Compliance by Constraint Position and Quantity
  11. Conclusions
  12. Appendix: Additional Related Works Considerations
  13. Appendix: Prompt Constraints Parameters
  14. Appendix: Constraints Distributions Charts
  15. Appendix: Evaluation Functions
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Distribution of constraint appearances by list size. Each bar shows the total number of times each constraint appears in prompts with a given number of constraints, illustrating how constraint usage varies across different list sizes in the dataset.
  • Figure 2: Constraint compliance by rank.
  • Figure 3: Prompt compliance by number of constraints.
  • Figure 4: Constraints frequency by product/service description.
  • Figure 5: Constraints frequency by task.