Think Straight, Stop Smart: Structured Reasoning for Efficient Multi-Hop RAG

TL;DR

The paper tackles the efficiency gap in on-device multi-hop retrieval-augmented generation by identifying predictable token waste and unreliable termination in existing methods. It proposes Think Straight, Stop Smart (TSSS), which combines template-based reasoning with KV-Cache to reuse recurring prefixes and a retriever-based terminator to deterministically stop when sub-queries repeat, effectively decoupling reasoning from stopping. KV-Cache reduces decoding complexity from $O(T^2 d^2) + O(T^3 d)$ to $O(T d^2 + T^2 d)$, enabling substantial token savings while maintaining accuracy. Empirical results on HotpotQA, 2WikiMultiHopQA, and MuSiQue show state-of-the-art EM and ACC_L with competitive latency and fewer tokens than competing RAG-CoT methods, indicating strong practical value for privacy-preserving, on-device reasoning. The approach enables accurate, efficient, on-device multi-hop reasoning without extra training, supporting latency-sensitive and private applications.

Abstract

Multi-hop retrieval-augmented generation (RAG) is a promising strategy for complex reasoning, yet existing iterative prompting approaches remain inefficient. They often regenerate predictable token sequences at every step and rely on stochastic stopping, leading to excessive token usage and unstable termination. We propose TSSS (Think Straight, Stop Smart), a structured multi-hop RAG framework designed for efficiency. TSSS introduces (i) a template-based reasoning that caches recurring prefixes and anchors sub-queries to the main question, reducing token generation cost while promoting stable reasoning, and (ii) a retriever-based terminator, which deterministically halts reasoning once additional sub-queries collapse into repetition. This separation of structured reasoning and termination control enables both faster inference and more reliable answers. On HotpotQA, 2WikiMultiHop, and MuSiQue, TSSS achieves state-of-the-art accuracy and competitive efficiency among RAG-CoT approaches, highlighting its effectiveness in efficiency-constrained scenarios such as on-device inference.

Figures (4)

• Figure 1: Comparison of multi-hop reasoning processes. (a) The baseline method (e.g., Self-Ask) suffers from inefficiencies such as repetitive prefix generation and duplicated queries, which increase token generation usage and may lead to incorrect answers. (b) Our TSSS framework addresses these issues through a structured reasoning template that reuses recurring token sequences and anchors sub-queries to the main question, together with a retriever-based terminator that halts reasoning once further queries collapse into repetition. This combination reduces token usage while maintaining accuracy, resulting in faster and more reliable inference.
• Figure 2: Performance vs. efficiency on the MuSiQue dataset.TSSS achieves a superior performance-efficiency trade-off, with the highest EM score and the fewest tokens.
• Figure 3: Effect of retriever-based terminator.TSSS proves efficient and stable termination mechanism by stopping fewer retrieval iterations than the other baselines.
• Figure 4: Performance–efficiency trade-off of different methods on three datasets.TSSS achieves the highest performance (EM) with fewer tokens than other baselines, demonstrating a superior balance of performance and efficiency.