Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RIFT: Reordered Instruction Following Testbed To Evaluate Instruction Following in Singular Multistep Prompt Structures

Andrew Jaffe, Noah Reicin, Jinho D. Choi

TL;DR

RIFT exposes a fundamental vulnerability in current LLMs: instruction-following quality is highly sensitive to prompt topology. By holding content constant and contrasting linear versus non-linear traversal on Jeopardy!-style QA, it demonstrates that structural discontinuity can collapse accuracy even in large, reasoning-tuned models, with errors primarily due to instruction-order violations and semantic drift. The framework uses a graph-based prompt formulation, a fixed system prompt, and an LLM-based evaluator with a structural-adherence metric to isolate structural effects. The work highlights the gap between nominal context window sizes and effective instruction-following capacity for non-sequential tasks, and it motivates architectural and training directions such as explicit state-tracking and graph-aware attention to enable robust non-linear instruction following in real-world workflows and multi-agent systems.

Abstract

Large Language Models (LLMs) are increasingly relied upon for complex workflows, yet their ability to maintain flow of instructions remains underexplored. Existing benchmarks conflate task complexity with structural ordering, making it difficult to isolate the impact of prompt topology on performance. We introduce RIFT, Reordered Instruction Following Testbed, to assess instruction following by disentangling structure from content. Using rephrased Jeopardy! question-answer pairs, we test LLMs across two prompt structures: linear prompts, which progress sequentially, and jumping prompts, which preserve identical content but require non-sequential traversal. Across 10,000 evaluations spanning six state-of-the-art open-source LLMs, accuracy dropped by up to 72% under jumping conditions (compared to baseline), revealing a strong dependence on positional continuity. Error analysis shows that approximately 50% of failures stem from instruction-order violations and semantic drift, indicating that current architectures internalize instruction following as a sequential pattern rather than a reasoning skill. These results reveal structural sensitivity as a fundamental limitation in current architectures, with direct implications for applications requiring non-sequential control flow such as workflow automation and multi-agent systems.

RIFT: Reordered Instruction Following Testbed To Evaluate Instruction Following in Singular Multistep Prompt Structures

TL;DR

RIFT exposes a fundamental vulnerability in current LLMs: instruction-following quality is highly sensitive to prompt topology. By holding content constant and contrasting linear versus non-linear traversal on Jeopardy!-style QA, it demonstrates that structural discontinuity can collapse accuracy even in large, reasoning-tuned models, with errors primarily due to instruction-order violations and semantic drift. The framework uses a graph-based prompt formulation, a fixed system prompt, and an LLM-based evaluator with a structural-adherence metric to isolate structural effects. The work highlights the gap between nominal context window sizes and effective instruction-following capacity for non-sequential tasks, and it motivates architectural and training directions such as explicit state-tracking and graph-aware attention to enable robust non-linear instruction following in real-world workflows and multi-agent systems.

Abstract

Large Language Models (LLMs) are increasingly relied upon for complex workflows, yet their ability to maintain flow of instructions remains underexplored. Existing benchmarks conflate task complexity with structural ordering, making it difficult to isolate the impact of prompt topology on performance. We introduce RIFT, Reordered Instruction Following Testbed, to assess instruction following by disentangling structure from content. Using rephrased Jeopardy! question-answer pairs, we test LLMs across two prompt structures: linear prompts, which progress sequentially, and jumping prompts, which preserve identical content but require non-sequential traversal. Across 10,000 evaluations spanning six state-of-the-art open-source LLMs, accuracy dropped by up to 72% under jumping conditions (compared to baseline), revealing a strong dependence on positional continuity. Error analysis shows that approximately 50% of failures stem from instruction-order violations and semantic drift, indicating that current architectures internalize instruction following as a sequential pattern rather than a reasoning skill. These results reveal structural sensitivity as a fundamental limitation in current architectures, with direct implications for applications requiring non-sequential control flow such as workflow automation and multi-agent systems.
Paper Structure (36 sections, 4 equations, 9 figures, 2 tables)

This paper contains 36 sections, 4 equations, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Instruction Tuning and Alignment
  4. Reasoning-Oriented Prompting
  5. Multi-Instruction Robustness and Order Effects
  6. Benchmark Limitations and Structural Gaps
  7. Approach
  8. Overview
  9. Conceptual Framework
  10. Prompt Construction
  11. System Prompt
  12. User Prompt
  13. Evaluation
  14. Rationale
  15. Experiments
  16. ...and 21 more sections

Figures (9)

  • Figure 1: Visualization of the out-of-order flow of a jumping prompt
  • Figure 2: Example prompt configuration.
  • Figure 3: Accuracy as number of tokens per prompt increases
  • Figure 4: Percentage of Incorrect Answers Caused by Answering the Wrong Question
  • Figure 5: Question-level accuracy compared to the number of questions previously answered (i.e. depth into the prompt)
  • ...and 4 more figures