Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models

TL;DR

This work investigates why MCQA prompts are highly sensitive to prompt structure, showing that placing the context before the question and options (CQO) yields substantial performance advantages over the reverse (QOC). Through controlled experiments across 21 decoder-only LLMs and four datasets, the authors identify causal attention as the root cause: in QOC, option tokens are masked from attending to context, creating an information bottleneck. Encoder-based models do not exhibit the gap, reinforcing the causal-masking explanation. The study further demonstrates interventions—attention pruning, activation patching, and option repetition—that can degrade CQO or improve QOC, offering practical guidance to manage prompt sensitivity and insights into the internal mechanisms of prompt structure effects.

Abstract

Large language models exhibit surprising sensitivity to the structure of the prompt, but the mechanisms underlying this sensitivity remain poorly understood. In this work, we conduct an in-depth investigation on a striking case: in multiple-choice question answering, placing context before the questions and options (CQO) outperforms the reverse order (QOC) by over 14%p, consistently over a wide range of models and datasets. Through systematic architectural analysis, we identify causal attention as the core mechanism: in QOC prompts, the causal mask prevents option tokens from attending to context, creating an information bottleneck where context becomes invisible to options.

Figures (7)

• Figure 1: Performance gap between CQO and QOC. We measure the average accuracies of 21 decoder-only LLMs on 4 different datasets, when prompted in two distinct structures; CQO (context-question-option) and QOC (question-option-context).
• Figure 2: Decoder vs. Encoder attention. In QOC (Question$\rightarrow$Options$\rightarrow$Context), causal masking prevents decoder models from attending to the context while selecting among options, so they often answer from option priors rather than evidence. Encoder models use bidirectional attention and can condition on the context when scoring the options.
• Figure 3: Hypothesis 1: Effect of instruction tuning and in-context learning. (a) We compare the performance gaps between base and instruct models, and find they remains remarkably consistent. (b) Few-shot prompting yields marginal gains. These results suggest that the friendly formatting is not the primary driver.
• Figure 4: Hypothesis 2: Option recall analysis. To investigate "forgetting" the options due to the long context intervention, we evaluated option recall accuracy. Results show that accuracy is consistently higher than CQO, indicating that the models retain option information and ruling out memory loss as the primary cause.
• Figure 5: Hypothesis 3, Exp# 1: Architecture comparison. (a) Decoder-only LLMs show a large gap. In contrast, encoder-only and encoder-decoder LLMs have a minimal accuracy gap, confirming that the causal mask is the primary factor. (b) For decoder-only LLMs, QOC performance drops to nearly QO, indicating that the information inside the context is ignored.