Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Instruction Gap: LLMs get lost in Following Instruction

Vishesh Tripathi, Uday Allu, Biddwan Ahmed

TL;DR

The paper investigates the instruction gap in large language models within enterprise Retrieval-Augmented Generation tasks, evaluating 13 leading LLMs on instruction compliance and response quality. It introduces a dedicated instruction-following benchmark, a rule-based violation detector, and an LLM-as-a-Judge framework to quantify adherence and output accuracy. The results reveal substantial cross-model variance, with GPT-5 variants delivering the best instruction compliance and strong accuracy, while Claude-4-Sonnet excels as a non-reasoning baseline; reasoning-enabled models show mixed gains in accuracy. The findings underscore a need to treat instruction adherence as a distinct deployment factor, offer practical benchmarks for enterprise selection, and point to architectural or prompting strategies that can reduce the instruction gap in production systems.

Abstract

Large Language Models (LLMs) have shown remarkable capabilities in natural language understanding and generation, yet their deployment in enterprise environments reveals a critical limitation: inconsistent adherence to custom instructions. This study presents a comprehensive evaluation of 13 leading LLMs across instruction compliance, response accuracy, and performance metrics in realworld RAG (Retrieval-Augmented Generation) scenarios. Through systematic testing with samples and enterprise-grade evaluation protocols, we demonstrate that instruction following varies dramatically across models, with Claude-Sonnet-4 and GPT-5 achieving the highest results. Our findings reveal the "instruction gap" - a fundamental challenge where models excel at general tasks but struggle with precise instruction adherence required for enterprise deployment. This work provides practical insights for organizations deploying LLM-powered solutions and establishes benchmarks for instruction-following capabilities across major model families.

The Instruction Gap: LLMs get lost in Following Instruction

TL;DR

The paper investigates the instruction gap in large language models within enterprise Retrieval-Augmented Generation tasks, evaluating 13 leading LLMs on instruction compliance and response quality. It introduces a dedicated instruction-following benchmark, a rule-based violation detector, and an LLM-as-a-Judge framework to quantify adherence and output accuracy. The results reveal substantial cross-model variance, with GPT-5 variants delivering the best instruction compliance and strong accuracy, while Claude-4-Sonnet excels as a non-reasoning baseline; reasoning-enabled models show mixed gains in accuracy. The findings underscore a need to treat instruction adherence as a distinct deployment factor, offer practical benchmarks for enterprise selection, and point to architectural or prompting strategies that can reduce the instruction gap in production systems.

Abstract

Large Language Models (LLMs) have shown remarkable capabilities in natural language understanding and generation, yet their deployment in enterprise environments reveals a critical limitation: inconsistent adherence to custom instructions. This study presents a comprehensive evaluation of 13 leading LLMs across instruction compliance, response accuracy, and performance metrics in realworld RAG (Retrieval-Augmented Generation) scenarios. Through systematic testing with samples and enterprise-grade evaluation protocols, we demonstrate that instruction following varies dramatically across models, with Claude-Sonnet-4 and GPT-5 achieving the highest results. Our findings reveal the "instruction gap" - a fundamental challenge where models excel at general tasks but struggle with precise instruction adherence required for enterprise deployment. This work provides practical insights for organizations deploying LLM-powered solutions and establishes benchmarks for instruction-following capabilities across major model families.
Paper Structure (24 sections, 2 figures, 1 table)

This paper contains 24 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Contributions
  3. Background and Related Work
  4. Instruction Following in Large Language Models
  5. RAG Systems and Enterprise Deployment
  6. LLM Evaluation Methodologies
  7. Instruction Gap and Compliance Research
  8. Experiment
  9. Dataset
  10. Evaluation Metrics
  11. Instruction Violation Assessment
  12. Response Quality Evaluation
  13. LLM-as-a-Judge Validation
  14. Experimental Setup and Model Configuration
  15. Results
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Performance comparison of large language models on enterprise RAG scenarios across instruction following and response accuracy metrics.
  • Figure 2: Experimental Workflow Overview.