Failure Modes in Multi-Hop QA: The Weakest Link Law and the Recognition Bottleneck

TL;DR

The "Weakest Link Law" is established: multi-hop reasoning performance collapses to the performance level of the least visible evidence, and it is demonstrated that "thinking" models that utilize System-2 reasoning, effectively locate and integrate the required information, matching gold-only baselines even in noisy, long-context settings.

Abstract

Despite scaling to massive context windows, Large Language Models (LLMs) struggle with multi-hop reasoning due to inherent position bias, which causes them to overlook information at certain positions. Whether these failures stem from an inability to locate evidence (recognition failure) or integrate it (synthesis failure) is unclear. We introduce Multi-Focus Attention Instruction (MFAI), a semantic probe to disentangle these mechanisms by explicitly steering attention towards selected positions. Across 5 LLMs on two multi-hop QA tasks (MuSiQue and NeoQA), we establish the "Weakest Link Law": multi-hop reasoning performance collapses to the performance level of the least visible evidence. Crucially, this failure is governed by absolute position rather than the linear distance between facts (performance variance $<3%$). We further identify a duality in attention steering: while matched MFAI resolves recognition bottlenecks, improving accuracy by up to 11.5% in low-visibility positions, misleading MFAI triggers confusion in real-world tasks but is successfully filtered in synthetic tasks. Finally, we demonstrate that "thinking" models that utilize System-2 reasoning, effectively locate and integrate the required information, matching gold-only baselines even in noisy, long-context settings.

Figures (8)

• Figure 1: Experimental setups shown in vertical columns. (a) Spread Test: Gold documents (orange) are placed in the Middle bucket, and the distance between them varies. (b) Cross Test: Gold documents are split between the Beginning and Tail buckets, maintaining the same local index $k$. (c) Unmatched Logic: This illustrates how a misleading (unmatched) instruction points to non-gold documents (red) in an unselected bucket by mirroring the local index $k$ of the true gold documents.
• Figure 2: Model performance across various MFAI (No MFAI, Matched, and Unmatched) on the MuSiQue and NeoQA datasets within the Spread Test. Bars represent the average accuracy across five inter-gold-document distances within each positional bucket. Accuracy values are presented as percentages, with No MFAI in white, Matched in blue, and Unmatched in red.
• Figure 3: Model performance across various attention instruction conditions (No MFAI, Matched, Unmatched) on the MuSiQue and NeoQA datasets within the Cross Test. Bars represent the average accuracy across six local indexes within the selected pair of positional buckets. Accuracy values are presented as percentages, with No MFAI in white, Matched in blue, and Unmatched in red.
• Figure 4: Model performance across various inter-gold-document distances in each positional bucket. The solid line represents the No MFAI accuracy, whereas the dash line represents the matched instruction accuracy.
• Figure 5: Layer-wise Heatmap.