Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Look It Up: Analysing Internal Web Search Capabilities of Modern LLMs

Sahil Kale

TL;DR

This paper probes whether frontier LLMs know when to invoke inbuilt web search and how retrieval affects accuracy and calibration. It introduces a two-split benchmark (static and dynamic) using GPT-5-mini and Claude Haiku 4.5 to externally evaluate search decisions without access to internal states. Key findings show that web search improves static factual accuracy but worsens confidence calibration, while dynamic time-sensitive queries achieve limited accuracy due to query formulation; costs remain modest and selective invocation is beneficial. The work provides deployment guidance and highlights the need to improve internal decision mechanisms, query formulation, and evidence handling, while releasing code and data for transparent evaluation.

Abstract

Modern large language models integrate web search to provide real-time answers, yet it remains unclear whether they are efficiently calibrated to use search when it is actually needed. We introduce a benchmark evaluating both the necessity and effectiveness of web access across commercial models with no access to internal states or parameters. The dataset includes a static split of 783 temporally anchored questions answerable from pre-cutoff knowledge, aimed at testing whether models invoke search based on low internal confidence, and a dynamic split of 288 post-cutoff queries designed to test whether models recognise when search is required and retrieve updated information. Web access substantially improves static accuracy for GPT-5-mini and Claude Haiku 4.5, though confidence calibration worsens. On dynamic queries, both models frequently invoke search yet remain below 70 percent accuracy due to weak query formulation. Costs per accuracy-improving call remain low, but returns diminish once initial retrieval fails. Selective invocation helps, but models become overconfident and inconsistent after search. Overall, built-in web search meaningfully improves factual accuracy and can be invoked selectively, yet models remain overconfident, skip retrieval when it is essential, and falter once initial search queries underperform. Taken together, internal web search works better as a good low-latency verification layer than a reliable analytical tool, with clear room for improvement.

Look It Up: Analysing Internal Web Search Capabilities of Modern LLMs

TL;DR

This paper probes whether frontier LLMs know when to invoke inbuilt web search and how retrieval affects accuracy and calibration. It introduces a two-split benchmark (static and dynamic) using GPT-5-mini and Claude Haiku 4.5 to externally evaluate search decisions without access to internal states. Key findings show that web search improves static factual accuracy but worsens confidence calibration, while dynamic time-sensitive queries achieve limited accuracy due to query formulation; costs remain modest and selective invocation is beneficial. The work provides deployment guidance and highlights the need to improve internal decision mechanisms, query formulation, and evidence handling, while releasing code and data for transparent evaluation.

Abstract

Modern large language models integrate web search to provide real-time answers, yet it remains unclear whether they are efficiently calibrated to use search when it is actually needed. We introduce a benchmark evaluating both the necessity and effectiveness of web access across commercial models with no access to internal states or parameters. The dataset includes a static split of 783 temporally anchored questions answerable from pre-cutoff knowledge, aimed at testing whether models invoke search based on low internal confidence, and a dynamic split of 288 post-cutoff queries designed to test whether models recognise when search is required and retrieve updated information. Web access substantially improves static accuracy for GPT-5-mini and Claude Haiku 4.5, though confidence calibration worsens. On dynamic queries, both models frequently invoke search yet remain below 70 percent accuracy due to weak query formulation. Costs per accuracy-improving call remain low, but returns diminish once initial retrieval fails. Selective invocation helps, but models become overconfident and inconsistent after search. Overall, built-in web search meaningfully improves factual accuracy and can be invoked selectively, yet models remain overconfident, skip retrieval when it is essential, and falter once initial search queries underperform. Taken together, internal web search works better as a good low-latency verification layer than a reliable analytical tool, with clear room for improvement.

Paper Structure

This paper contains 29 sections, 8 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Web-Augmented LLMs
  4. Browsing and Tool-Use Benchmarks
  5. Methodology
  6. Static Split
  7. Dataset
  8. Evaluation and Metrics
  9. Dynamic Split
  10. Dataset
  11. Evaluation and Metrics
  12. Experimental Setup
  13. Analysis and Result Discussion
  14. How effective is web search in improving LLM accuracy?
  15. Does web search change factual outcomes when invoked?
  16. ...and 14 more sections

Figures (8)

  • Figure 1: Evaluation pipeline for assessing internal web search behaviour in LLMs
  • Figure 2: Accuracy by number of web search calls, with 95% confidence intervals, evaluated under the web-search enabled condition on the static split
  • Figure 3: Proportion of outcome transitions for both models with and without web search
  • Figure 4: Outcome transitions binned by self-reported confidence for both models with and without web search
  • Figure 5: Accuracy versus web search rate under confidence-gated web search, compared to random gating, always-retrieve, LLM-only baselines, and the oracle upper bound
  • ...and 3 more figures