LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services

TL;DR

LocalSearchBench provides the first large-scale, real-world benchmark for agentic search in local life services, addressing complex, multi-hop tasks across dining, shopping, accommodation, travel, and healthcare. The authors construct a 150k-merchant database, a 300-task multi-hop QA suite, and a LocalRAG-enabled evaluation environment (LocalPlayground) to test LRMs in a realistic local domain. Evaluations across ten LRMs reveal that current systems struggle with correctness, completeness, and faithfulness, even with web search augmentation, highlighting the need for domain-specific data, tools, and training. The work delivers open-source benchmark assets and tooling to catalyze progress in domain-oriented agentic search and cross-source reasoning for local services.

Abstract

Recent advances in large reasoning models (LRMs) have enabled agentic search systems to perform complex multi-step reasoning across multiple sources. However, most studies focus on general information retrieval and rarely explores vertical domains with unique challenges. In this work, we focus on local life services and introduce LocalSearchBench, which encompass diverse and complex business scenarios. Real-world queries in this domain are often ambiguous and require multi-hop reasoning across merchants and products, remaining challenging and not fully addressed. As the first comprehensive benchmark for agentic search in local life services, LocalSearchBench includes over 150,000 high-quality entries from various cities and business types. We construct 300 multi-hop QA tasks based on real user queries, challenging agents to understand questions and retrieve information in multiple steps. We also developed LocalPlayground, a unified environment integrating multiple tools for agent interaction. Experiments show that even state-of-the-art LRMs struggle on LocalSearchBench: the best model (DeepSeek-V3.1) achieves only 34.34% correctness, and most models have issues with completeness (average 77.33%) and faithfulness (average 61.99%). This highlights the need for specialized benchmarks and domain-specific agent training in local life services. Code, Benchmark, and Leaderboard are available at localsearchbench.github.io.

Figures (14)

• Figure 1: Illustration of different types of QA tasks
• Figure 2: Workflow of Local Merchant Database Construction
• Figure 3: Word clouds showing the landmark locations across 3 major Chinese cities.
• Figure 4: City distribution and category distribution of 150,031 merchant data across three cities.
• Figure 5: Geographical distribution heatmaps of 150,031 merchant data across three cities.