How Far are LLMs from Real Search? A Comprehensive Study on Efficiency, Completeness, and Inherent Capabilities

TL;DR

This work tackles the challenge of enabling efficient yet complete problem-solving with LLMs by integrating learning into traditional search. It introduces SeaL, a framework that uses learning-driven components—Direct Solution Generation, State Decomposition, State Validity Checking, and Learning-guided Ranking—to dramatically reduce search space while maintaining completeness. To guarantee completeness, SeaL-C adds learning-guided complete state decomposition and two-phase ranking, achieving rigorous results across planning tasks such as Game of 24, Mini Crosswords, and Blocksworld. The study also investigates whether LLMs can learn to search independently, finding that current models struggle with efficient self-search, but SeaL-inspired strategies substantially improve performance. Overall, the bidirectional synergy between search and learning demonstrated here has practical implications for real-world decision-making tasks and motivates further work on self-search in LLMs and broader applicability to multi-modal systems.

Abstract

Search plays a fundamental role in problem-solving across various domains, with most real-world decision-making problems being solvable through systematic search. Drawing inspiration from recent discussions on search and learning, we systematically explore the complementary relationship between search and Large Language Models (LLMs) from three perspectives. First, we analyze how learning can enhance search efficiency and propose Search via Learning (SeaL), a framework that leverages LLMs for effective and efficient search. Second, we further extend SeaL to SeaL-C to ensure rigorous completeness during search. Our evaluation across three real-world planning tasks demonstrates that SeaL achieves near-perfect accuracy while reducing search spaces by up to 99.1% compared to traditional approaches. Finally, we explore how far LLMs are from real search by investigating whether they can develop search capabilities independently. Our analysis reveals that while current LLMs struggle with efficient search in complex problems, incorporating systematic search strategies significantly enhances their problem-solving capabilities. These findings not only validate the effectiveness of our approach but also highlight the need for improving LLMs' search abilities for real-world applications.

Figures (9)

• Figure 1: PR (%) and SS of existing searches across various problem difficulties using GPT-4o-mini.
• Figure 2: $\textnormal{SeaL}$ intergrating learning into search with LLMs: (1) Direct solution generation, (2) State validity checking, (3) Learning-guided state ranking.
• Figure 3: Impact of problem difficulty on search completeness in Game of 24 and Blocksworld with GPT-4o-Mini.
• Figure 4: Impact of SS in $\textnormal{SeaL}$ using GPT-4o-mini.
• Figure 5: Illustrative examples of Observation 1 in Sec. \ref{['sec:preliminary_results_analysis']}. This figure presents three examples of calculating $24$ from intermediate steps in the Game of 24 task. The green answer represents a correct equation that results in $24$, whereas the red answer represents an incorrect equation that does not equal $24$. We observe that LLMs perform well in solving simple tasks in one step, such as $4 \times 6 = 24$ and $(10 - 6) \times 5 + 4 = 24$, but struggle with more complex tasks in a single step (e.g., the third example, where the model fails to find a solution using numbers $5$, $5$, $8$, and $10$).

Theorems & Definitions (1)

• Definition 4.1: Search Completeness