Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Search, Do not Guess: Teaching Small Language Models to Be Effective Search Agents

Yizhou Liu, Qi Sun, Yulin Chen, Siyue Zhang, Chen Zhao

Abstract

Agents equipped with search tools have emerged as effective solutions for knowledge-intensive tasks. While Large Language Models (LLMs) exhibit strong reasoning capabilities, their high computational cost limits practical deployment for search agents. Consequently, recent work has focused on distilling agentic behaviors from LLMs into Small Language Models (SLMs). Through comprehensive evaluation on complex multi-hop reasoning tasks, we find that despite possessing less parametric knowledge, SLMs invoke search tools less frequently and are more prone to hallucinations. To address this issue, we propose \policy, a lightweight fine-tuning approach that explicitly trains SLMs to reliably retrieve and generate answers grounded in retrieved evidence. Compared to agent distillation from LLMs, our approach improves performance by 17.3 scores on Bamboogle and 15.3 scores on HotpotQA, achieving LLM-level results across benchmarks. Our further analysis reveals that adaptive search strategies in SLMs often degrade performance, highlighting the necessity of consistent search behavior for reliable reasoning.

Search, Do not Guess: Teaching Small Language Models to Be Effective Search Agents

Abstract

Agents equipped with search tools have emerged as effective solutions for knowledge-intensive tasks. While Large Language Models (LLMs) exhibit strong reasoning capabilities, their high computational cost limits practical deployment for search agents. Consequently, recent work has focused on distilling agentic behaviors from LLMs into Small Language Models (SLMs). Through comprehensive evaluation on complex multi-hop reasoning tasks, we find that despite possessing less parametric knowledge, SLMs invoke search tools less frequently and are more prone to hallucinations. To address this issue, we propose \policy, a lightweight fine-tuning approach that explicitly trains SLMs to reliably retrieve and generate answers grounded in retrieved evidence. Compared to agent distillation from LLMs, our approach improves performance by 17.3 scores on Bamboogle and 15.3 scores on HotpotQA, achieving LLM-level results across benchmarks. Our further analysis reveals that adaptive search strategies in SLMs often degrade performance, highlighting the necessity of consistent search behavior for reliable reasoning.

Paper Structure

This paper contains 49 sections, 1 equation, 3 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Task Formulation
  3. Evaluating Vanilla and Distilled SLM Agents
  4. Vanilla Performance
  5. Agentic Distillation
  6. Setups.
  7. Results.
  8. Distillation with Always-Search Policy
  9. Experiment Setup
  10. Fine-tuning Techniques.
  11. Evaluation.
  12. Results
  13. The performance gap narrows after ASP.
  14. Consistent search tool calling.
  15. Robustness to noisy retrieval.
  16. ...and 34 more sections

Figures (3)

  • Figure 1: Left: With Always-Search Policy, the distilled SLM significantly narrows the performance gap with the teacher model. Right: SLMs suffer from adaptive search and ASP is the most effective policy.
  • Figure 2: Scaling of agentic search performance. Small models trail behind in both performance and retrieval capability on HotpotQA.
  • Figure 3: Confidence probing results. (a) illustrates the full sample distribution on a log scale; (b) zooms into high-confidence bins (0.5--1.0).