When Models Can't Follow: Testing Instruction Adherence Across 256 LLMs

TL;DR

The study addresses the challenge of reliably evaluating instruction adherence in contemporary LLMs. It introduces a compact, twenty-prompt diagnostic suite deployed via OpenRouter to 331 models, followed by a rigorous two-stage verification to ensure functional endpoints before assessing 256 verified models. Across 5,120 evaluations, the results reveal substantial variability in instruction-following abilities that do not scale cleanly with model size, and highlight distinct failure modes such as string-manipulation and strict format compliance. The framework provides a practical, low-cost diagnostic tool for practitioners to diagnose instruction adherence patterns and informs training strategies to improve constraint satisfaction and output formatting across diverse model families.

Abstract

Despite widespread deployment of Large Language Models, systematic evaluation of instruction-following capabilities remains challenging. While comprehensive benchmarks exist, focused assessments that quickly diagnose specific instruction adherence patterns are valuable. As newer models may be trained on existing benchmarks, novel evaluation approaches are needed to assess genuine capabilities rather than memorized performance. This paper presents a streamlined evaluation framework using twenty carefully designed prompts to assess LLM instruction-following across diverse task categories. We demonstrate this framework through a large-scale empirical study conducted on October 14, 2025, testing 256 verified working models from 331 available via OpenRouter. To ensure methodological rigor and prevent selection bias, we first verified each model's basic functionality before inclusion. Unlike large-scale benchmarks requiring extensive computational resources, our approach offers a practical diagnostic tool researchers and practitioners can readily apply. Our methodology builds upon verifiable instructions while introducing a compact test suite balancing comprehensiveness with efficiency. Each prompt targets distinct aspects of instruction following, including format compliance, content constraints, logical sequencing, and multi-step task execution. We evaluate models from major providers (OpenAI, Anthropic, Google, Meta, Mistral) and emerging implementations (Qwen, DeepSeek, community models), providing comparative performance analysis. Our findings reveal consistent failure modes and identify specific instruction types posing particular challenges. This work contributes both a practical evaluation tool and one of the most comprehensive empirical analyses of instruction-following capabilities across the contemporary LLM landscape.

Figures (3)

• Figure 1: Comprehensive instruction-following performance across verified models (Top 50). Heat map displaying binary pass/fail results for the top 50 performing models across all 20 diagnostic tests. Rows represent individual models ordered by overall performance. Columns represent the twenty diagnostic tasks (Tests 1-20). Green indicates successful instruction adherence; red indicates failure. The visualization reveals distinct performance clusters, with top models (e.g., qwen/qwen-plus-2025-07-28:thinking achieving 100% pass rate) demonstrating consistent success across most tests, while even high-performing models struggle with certain string manipulation tasks (Tests 1, 3, 5, 17).
• Figure 2: Test difficulty varies dramatically across instruction types. Horizontal bar chart displays pass rates for all 20 diagnostic tests, sorted from hardest (top) to easiest (bottom). Colors indicate difficulty levels: dark red ($<$10%, extremely hard), red (10-20%, very hard), orange (20-40%, hard), yellow (40-60%, medium), light green (60-80%, easy), and dark green ($>$80%, very easy). String manipulation tests (Tests 1, 3, 5, 17, 20) dominate the hardest category, while constraint compliance tests (Tests 2, 15, 9) achieve the highest pass rates. Notable findings: Test 5 (String Manipulation Chain) at 2.7% represents an extreme challenge for current models, while Test 13 (Selective Text Processing) at 80.1% demonstrates strong performance on targeted text operations.
• Figure 3: Instruction-following performance by model provider family. Bar chart displays average pass rates for the top 12 model providers (minimum 3 models each). x-ai models demonstrate the strongest instruction-following capabilities (79.3%), substantially outperforming the overall average (indicated by red dashed line). Traditional large providers like Google, OpenAI, and Qwen cluster near the average, while newer or specialized providers show more variable performance. Error bars would represent 95% confidence intervals if included. The substantial variation between providers (ranging from 33.3% to 79.3%) indicates that training methodologies and model architectures significantly impact instruction adherence capabilities.