UIS-Digger: Towards Comprehensive Research Agent Systems for Real-world Unindexed Information Seeking

TL;DR

UIS-QA is introduced, the first dedicated UIS benchmark, and UIS-Digger, a novel multi-agent framework that incorporates dual-mode browsing and enables simultaneous webpage searching and file parsing, which demonstrates the importance of proactive interaction with unindexed sources for effective and comprehensive information-seeking.

Abstract

Recent advancements in LLM-based information-seeking agents have achieved record-breaking performance on established benchmarks. However, these agents remain heavily reliant on search-engine-indexed knowledge, leaving a critical blind spot: Unindexed Information Seeking (UIS). This paper identifies and explores the UIS problem, where vital information is not captured by search engine crawlers, such as overlooked content, dynamic webpages, and embedded files. Despite its significance, UIS remains an underexplored challenge. To address this gap, we introduce UIS-QA, the first dedicated UIS benchmark, comprising 110 expert-annotated QA pairs. Notably, even state-of-the-art agents experience a drastic performance drop on UIS-QA (e.g., from 70.90 on GAIA and 46.70 on BrowseComp-zh to 24.55 on UIS-QA), underscoring the severity of the problem. To mitigate this, we propose UIS-Digger, a novel multi-agent framework that incorporates dual-mode browsing and enables simultaneous webpage searching and file parsing. With a relatively small $\sim$30B-parameter backbone LLM optimized using SFT and RFT training strategies, UIS-Digger sets a strong baseline at 27.27\%, outperforming systems integrating sophisticated LLMs such as O3 and GPT-4.1. This demonstrates the importance of proactive interaction with unindexed sources for effective and comprehensive information-seeking. Our work not only uncovers a fundamental limitation in current agent evaluation paradigms but also provides the first toolkit for advancing UIS research, defining a new and promising direction for robust information-seeking systems.

Figures (5)

• Figure 1: UIS problem. Previous information-seeking agents (bottom) focus primarily on indexed information and thus often fail to gather the evidence needed to answer complex queries, either rejecting to answer or generate hallucinations. In contrast, UIS agents (top) are equipped with additional tools and fine-tuned to excavate unindexed information, thus capable of interacting with websites deeply and solve UIS tasks reliably.
• Figure 2: UIS-Digger multi-agent system. Planner, web searcher, web surfer and file reader works together to solve UIS problems. The web surfer can switch between textual- and visual-mode to observe webpages and hence make next-step decisions. Zoom-in for better view.
• Figure 3: QA Pairs Construction Pipeline. (Left): Procedure for constructing QA pairs using real-world information. First, homepages potentially containing deep navigation structures and informative content are collected. UIS-Digger then explores these homepages to extract information from pages requiring multiple navigation steps. The collected information subsequently serves as context for query generation. (Right): Procedure for constructing QA pairs based on simulated webpages. We identify browsing actions that UIS-Digger struggles to perform, and generate webpages (along with a JSON database containing relevant statistics) that incorporate these actions. QA pairs are then generated using the information from the JSON database of these simulated webpages.
• Figure 4: Action analysis. (left): Search behaviors of UIS-Digger and three baseline methods. The pie charts show the proportions of cases where the agent successfully retrieves the root URL via search, and whether the root URL is subsequently accessed through crawling or browsing. (Right): Action frequency distributions of correct and incorrect cases for Pangu-38B UIS-Digger at different training stages. Zoom-in for best view.
• Figure 5: The UIS-QA score changing curve for UIS-Digger at different stages of training.