Over-Searching in Search-Augmented Large Language Models

TL;DR

This work examines over-searching in search-augmented LLMs, where invoking external retrieval can exceed its utility and introduce noise. It develops a formal framework and the Tokens Per Correctness (TPC) metric to quantify the cost-effectiveness of search, and introduces OverSearchQA to systematically study abstention and search efficiency across answerable and unanswerable queries. Key findings show that while search generally improves correctness on solvable questions, it harms abstention on unanswerable ones, with effects amplified by model complexity, noisy retrieval, and multi-turn interactions; negative evidence in retrieved results can aid abstention. The authors explore query- and retrieval-level mitigations and release OverSearchQA to spur further research into more efficient and reliable search-augmented LLMs, highlighting the need for approaches that better balance accuracy with abstention and cost.

Abstract

Search-augmented large language models (LLMs) excel at knowledge-intensive tasks by integrating external retrieval. However, they often over-search -- unnecessarily invoking search tool even when it does not improve response quality, which leads to computational inefficiency and hallucinations by incorporating irrelevant context. In this work, we conduct a systematic evaluation of over-searching across multiple dimensions, including query types, model categories, retrieval conditions, and multi-turn conversations. Our finding shows: (i) search generally improves answer accuracy on answerable queries but harms abstention on unanswerable ones; (ii) over-searching is more pronounced in complex reasoning models and deep research systems, is exacerbated by noisy retrieval, and compounds across turns in multi-turn conversations; and (iii) the composition of retrieved evidence is crucial, as the presence of negative evidence improves abstention. To quantify over-searching, we introduce Tokens Per Correctness (TPC), an evaluation metric that captures the performance-cost trade-off for search-augmented LLMs. Lastly, we investigate mitigation approaches at both the query and retrieval levels and release the OverSearchQA to foster continued research into efficient search-augmented LLMs.

Figures (9)

• Figure 1: Illustration of over-searching in a search-augmented LLM. The question asks about an unknown future event. Compared to the base model that correctly recognizes this and abstains, the search-augmented LLM initiates unnecessary searches, leading to extra cost and a potential incorrect answer attempt.
• Figure 2: Performance of o4-mini as maximum search turns increase from 0 to 19. Answer accuracy (on answerable queries) significantly improves from no search to one search, then peaks around 7 searches and plateaus. Abstention accuracy (on unanswerable queries) consistently degrades with more searches. Meanwhile, TPC rises monotonically, demonstrating over-searching: costs accumulate faster than correctness gains, as additional searches neither improve answer accuracy nor prevent abstention degradation.
• Figure 3: (a) Length distributions show similar token counts between answerable and unanswerable questions. (b) t-SNE visualization of question embeddings reveals substantial semantic overlap, demonstrating that answerable and unanswerable questions are semantically indistinguishable. Category-specific similarity breakdown is shown in Appendix Figure \ref{['fig:combined_embeddings_tsne']}. (c) Word clouds of answerable and unanswerable questions in OverSearchQA.
• Figure 4: Comparison of the same model family with different configurations: Base (GPT-4o-mini), Reason (o4-mini), and Deep Research (o4-mini-deep-research). Answer accuracy increases while abstention accuracy consistently degrades as configurations become more complex. TPC (shown in log scale) increases with search capabilities; Deep Research dramatically reaches 38.9k TPC -- over 221$\times$ compared to the base configuration.
• Figure 5: TPC breakdown by outcome categories. Abstention failure remains the most expensive behavior for most models.