When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs

TL;DR

The paper shows that explicit reasoning via chain-of-thought prompting can degrade instruction-following accuracy across two benchmarks (IFEval and ComplexBench). It introduces a constraint-attention metric to reveal reduced attention to instruction constraints during reasoning and provides four mitigation strategies—few-shot in-context learning, self-reflection, self-selective reasoning, and classifier-selective reasoning—with classifier-selective reasoning delivering the most consistent improvements. The work combines large-scale empirical evaluation with qualitative case studies and attention analyses to diagnose reasoning-induced failures. It offers a practical deployment guideline favoring selective reasoning to preserve instruction adherence while retaining reasoning benefits for complex tasks.

Abstract

Reasoning-enhanced large language models (RLLMs), whether explicitly trained for reasoning or prompted via chain-of-thought (CoT), have achieved state-of-the-art performance on many complex reasoning tasks. However, we uncover a surprising and previously overlooked phenomenon: explicit CoT reasoning can significantly degrade instruction-following accuracy. Evaluating 15 models on two benchmarks: IFEval (with simple, rule-verifiable constraints) and ComplexBench (with complex, compositional constraints), we consistently observe performance drops when CoT prompting is applied. Through large-scale case studies and an attention-based analysis, we identify common patterns where reasoning either helps (e.g., with formatting or lexical precision) or hurts (e.g., by neglecting simple constraints or introducing unnecessary content). We propose a metric, constraint attention, to quantify model focus during generation and show that CoT reasoning often diverts attention away from instruction-relevant tokens. To mitigate these effects, we introduce and evaluate four strategies: in-context learning, self-reflection, self-selective reasoning, and classifier-selective reasoning. Our results demonstrate that selective reasoning strategies, particularly classifier-selective reasoning, can substantially recover lost performance. To our knowledge, this is the first work to systematically expose reasoning-induced failures in instruction-following and offer practical mitigation strategies.

Figures (9)

• Figure 1: Constraint-attention trace examples for Qwen2.5-1.5B-Instruct across both datasets. From top to bottom, the plots show cases where reasoning leads to a TIE, LOSE, and WIN compared to not using reasoning. The vertical red dashed line marks the start of Thinking, and the green dashed line marks the start of the Answer during response generation.
• Figure 2: Within IFEval dataset, drop in constraint attention (Base – CoT) for WIN vs. LOSE cases across representative layers of the Qwen2.5-1.5B-Instruct model. On average, the drop is more severe in cases when reasoning loses, compared to not using reasoning.
• Figure 3: Visualization of instruction-following accuracy across models and methods on IFEval and ComplexBench.
• Figure 4: Constraint-attention trace examples in IFEval for Llama3.2-1B-Instruct across both datasets. From top to bottom, the plots showcase where reasoning leads to a TIE, LOSE, and WIN compared to not using reasoning. The vertical red dashed line marks the start of Thinking, and the green dashed line marks the start of the Answer during response generation.
• Figure 5: Constraint-attention trace examples in ComplexBench for Llama3.2-1B-Instruct across both datasets. From top to bottom, the plots show cases where reasoning leads to a TIE, LOSE, and WIN compared to not using reasoning. The vertical red dashed line marks the start of Thinking, and the green dashed line marks the start of the Answer during response generation.