Adaptive Retrieval helps Reasoning in LLMs -- but mostly if it's not used
Srijan Shakya, Anamaria-Roberta Hartl, Sepp Hochreiter, Korbinian Pöppel
TL;DR
The paper addresses the tendency of LLMs to falter on complex reasoning due to static knowledge by introducing an adaptive retrieval-augmented reasoning agent that uses on-demand retrieval within a CoT framework. It demonstrates that selectively querying an external knowledge base improves performance on challenging math benchmarks (notably +$6.4$pp on MATH-500) while static, non-targeted retrieval can harm reasoning, and even when retrieval is not used, the model can exhibit strong performance, indicating a meaningful metacognitive signal. The key contribution is showing retrieval's value lies in when and how it is used, not merely in the retrieved content, and that retrieval decisions scale with problem difficulty. This points to a general principle for building robust generative models: treat retrieval as a dynamic, agentic, metacognitive tool for uncertainty-aware knowledge integration, rather than a fixed augmentation.
Abstract
Large Language Models (LLMs) often falter in complex reasoning tasks due to their static, parametric knowledge, leading to hallucinations and poor performance in specialized domains like mathematics. This work explores a fundamental principle for enhancing generative models: treating retrieval as a form of dynamic in-context learning. We test an adaptive retrieval-augmented architecture where an LLM agent actively decides when to query an external knowledge base during its reasoning process. We compare this adaptive strategy against a standard Chain-of-Thought (CoT) baseline and a static retrieval approach on the GSM8K and MATH-500 benchmarks. Although our experiments show that static retrieval is inferior to CoT, the adaptive retrieval shows interesting behavior: While traces including retrieved results show slightly worse performance compared to CoT, traces that do not include retrieval actually perform better compared to CoT. This suggests that: (a) retrieval only rarely helps reasoning (we show a few counterexamples, e.g. using useful theorems) and (b) actively not using retrieval is indicative of good model performance. Furthermore, we find that the model scales its retrieval frequency with the difficulty of the problem, reinforcing that the decision to retrieve is a crucial metacognitive signal. The agent's ability to self-assess its knowledge and selectively engage with external information represents a key principle for building more robust and reliable generative models.