On the Paradoxical Interference between Instruction-Following and Task Solving

TL;DR

The paper reveals a paradox where adding self-evident constraints to instructions can degrade an LLM's core task-solving ability. It introduces SustainScore, a formal metric and automated framework to quantify robustness of task performance under constrained instructions across mathematics, multi-hop question answering, and code generation. Empirical results show substantial performance drops under constraints, with degradation persisting across constraint types and scales, particularly in code generation. Mechanistic analysis via a Constraint Attention Score suggests that excessive attention to constraints during generation contributes to failure, and preliminary findings indicate post-training strategies (e.g., RL-based alignment) can modulate this interference. The work highlights the need to evaluate task robustness under constraints for reliable alignment and provides a foundation for developing more robust constrained-task solvers.

Abstract

Instruction following aims to align Large Language Models (LLMs) with human intent by specifying explicit constraints on how tasks should be performed. However, we reveal a counterintuitive phenomenon: instruction following can paradoxically interfere with LLMs' task-solving capability. We propose a metric, SUSTAINSCORE, to quantify the interference of instruction following with task solving. It measures task performance drop after inserting into the instruction a self-evident constraint, which is naturally met by the original successful model output and extracted from it. Experiments on current LLMs in mathematics, multi-hop QA, and code generation show that adding the self-evident constraints leads to substantial performance drops, even for advanced models such as Claude-Sonnet-4.5. We validate the generality of the interference across constraint types and scales. Furthermore, we identify common failure patterns, and by investigating the mechanisms of interference, we observe that failed cases allocate significantly more attention to constraints compared to successful ones. Finally, we use SUSTAINSCORE to conduct an initial investigation into how distinct post-training paradigms affect the interference, presenting empirical observations on current alignment strategies. We will release our code and data to facilitate further research

Figures (6)

• Figure 1: LLMs may fail at a task merely by adding a self-evident constraint that is already being met in their originally successful, unconstrained outputs.
• Figure 2: An overview of the evaluation framework for computing SustainScore.
• Figure 3: Comparison of SustainScore and Constraint Satisfaction Rate as the number of constraints increases.
• Figure 4: Comparison of constraint attention scores between successful and failed generations.
• Figure 5: SustainScore for different post-training strategies within the same model family on the Math task.