IHEval: Evaluating Language Models on Following the Instruction Hierarchy

TL;DR

This paper addresses the gap in evaluating language models on their ability to follow an instruction hierarchy that prioritizes system over user inputs, history, and tool outputs. It introduces IHEval, a programmable benchmark with 3,538 examples across nine tasks and four input types, designed to assess performance under aligned, conflict, and reference settings. Across 13 models, the study finds substantial performance drops when instructions conflict, with open-source models often under 50% accuracy in resolving conflicts, while larger models show better but incomplete resilience to hierarchy. The results reveal that current LMs struggle to internalize the hierarchy, with sensitivity to instruction strictness and minimal gains from prompting, underscoring a need for dedicated training to optimize hierarchical instruction following and to improve safety and reliability in real-world deployments.

Abstract

The instruction hierarchy, which establishes a priority order from system messages to user messages, conversation history, and tool outputs, is essential for ensuring consistent and safe behavior in language models (LMs). Despite its importance, this topic receives limited attention, and there is a lack of comprehensive benchmarks for evaluating models' ability to follow the instruction hierarchy. We bridge this gap by introducing IHEval, a novel benchmark comprising 3,538 examples across nine tasks, covering cases where instructions in different priorities either align or conflict. Our evaluation of popular LMs highlights their struggle to recognize instruction priorities. All evaluated models experience a sharp performance decline when facing conflicting instructions, compared to their original instruction-following performance. Moreover, the most competitive open-source model only achieves 48% accuracy in resolving such conflicts. Our results underscore the need for targeted optimization in the future development of LMs.

Figures (17)

• Figure 1: Four categories of the instruction hierarchy and the corresponding priority orders of instructions. Conflict instructions are shown in red. Models are expected to follow the instruction with the higher priority.
• Figure 2: Results of mainstream LMs on IHEval. The reference setting represents original task performance without hierarchical inputs. We observe large performance drops when models face conflicting hierarchical instructions.
• Figure 3: IHEval covers four categories and nine tasks. Detailed examples and instructions are in Figures \ref{['fig:task_card_single_turn']}\ref{['fig:task_card_slack']}.
• Figure 4: The original data source, the evaluation metric, and the data size of each task.
• Figure 5: The trend of IHEval performance by model scale.