Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple, Self-Contained Directives

Henry Lim, Kwan Hui Lim

TL;DR

The work addresses whether instruction-tuned LLMs can faithfully execute simple, self-contained directives by isolating atomic instruction-following through controlled label-format prompts on MMLU and MMLU-Pro. It leverages four prompt variations (instruction presence, option content, and few-shot exemplars) across 25 models, using strict exact-match evaluation and label-fidelity analyses. Key findings show strong instruction-format bias, persistent failure to treat labels as interchangeable placeholders, and limited benefits from few-shot guidance, even for larger models or after instruction tuning. The results highlight a fundamental gap in current instruction-tuning paradigms and call for targeted training and evaluation approaches that explicitly optimize atomic instruction-following to ensure reliable behavior in deterministic tasks.

Abstract

Instruction-tuned large language models (IT-LLMs) exhibit strong zero-shot reasoning, yet their ability to execute simple, self-contained instructions remains underexplored, despite this being foundational to complex instruction-following. We evaluate 20 IT-LLMs on modified MMLU and MMLU-Pro benchmarks, by systematically varying the format of option labels (alphabetic, numeric, Roman) while keeping their meaning identical under four paradigms, namely: (1) With explicit instructions, label changes cause large performance shifts (e.g., -30.45\% for Roman vs. numeric), revealing instruction-format bias. (2) Without instructions, performance drops further (up to -10.84\%) and label sensitivity intensifies, underscoring the role of explicit guidance. (3) When option contents are removed, models fail random-choice baselines except with numeric labels, suggesting weak adherence to atomic directives. (4) Three-shot exemplars yield no significant gains in robustness or fidelity, and generation analyses show persistent label errors, especially for non-numeric formats. Across model sizes, larger LLMs achieve higher accuracy but remain inconsistent in instruction adherence. These results expose the insufficiencies of current instruction-tuning paradigms and highlight the need for evaluation methods and training strategies that explicitly target atomic instruction-following.

The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple, Self-Contained Directives

TL;DR

The work addresses whether instruction-tuned LLMs can faithfully execute simple, self-contained directives by isolating atomic instruction-following through controlled label-format prompts on MMLU and MMLU-Pro. It leverages four prompt variations (instruction presence, option content, and few-shot exemplars) across 25 models, using strict exact-match evaluation and label-fidelity analyses. Key findings show strong instruction-format bias, persistent failure to treat labels as interchangeable placeholders, and limited benefits from few-shot guidance, even for larger models or after instruction tuning. The results highlight a fundamental gap in current instruction-tuning paradigms and call for targeted training and evaluation approaches that explicitly optimize atomic instruction-following to ensure reliable behavior in deterministic tasks.

Abstract

Instruction-tuned large language models (IT-LLMs) exhibit strong zero-shot reasoning, yet their ability to execute simple, self-contained instructions remains underexplored, despite this being foundational to complex instruction-following. We evaluate 20 IT-LLMs on modified MMLU and MMLU-Pro benchmarks, by systematically varying the format of option labels (alphabetic, numeric, Roman) while keeping their meaning identical under four paradigms, namely: (1) With explicit instructions, label changes cause large performance shifts (e.g., -30.45\% for Roman vs. numeric), revealing instruction-format bias. (2) Without instructions, performance drops further (up to -10.84\%) and label sensitivity intensifies, underscoring the role of explicit guidance. (3) When option contents are removed, models fail random-choice baselines except with numeric labels, suggesting weak adherence to atomic directives. (4) Three-shot exemplars yield no significant gains in robustness or fidelity, and generation analyses show persistent label errors, especially for non-numeric formats. Across model sizes, larger LLMs achieve higher accuracy but remain inconsistent in instruction adherence. These results expose the insufficiencies of current instruction-tuning paradigms and highlight the need for evaluation methods and training strategies that explicitly target atomic instruction-following.
Paper Structure (22 sections, 1 figure, 8 tables)

This paper contains 22 sections, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Model Selection and Decoding
  5. Datasets
  6. Prompt Setups
  7. Evaluation Metrics
  8. Analysis of Generated Answers
  9. Results and Analysis
  10. Main Results: Atomic Instruction Failure in Instruction-Tuned LLMs
  11. Instruction-Tuning Benefits and Limitations
  12. A Fundamental Failure: Instruction Following Without Content
  13. The Ineffectiveness of Few-Shot Learning
  14. Analysis of Generated Answers: Label Fidelity and Bias
  15. Label Fidelity
  16. ...and 7 more sections

Figures (1)

  • Figure 1: Overview of experimental setups. Each experiment systematically manipulates components of multiple-choice question prompts - instructions, option labels, option content, and few-shot examples—to test how model performance and response adherence vary under different input configurations. Experiment 1 includes all components; Experiment 2 removes the explicit instruction to test whether models can infer the task; Experiment 3 removes option content to approximate a random-guessing baseline; and Experiment 4 adds few-shot examples to evaluate whether in-context learning mitigates instruction-following failures. All experiments are repeated across three sets of option labels: alphabetical (A/B/C/D), numerical (1/2/3/4), and Roman (I/II/III/IV).