GISA: A Benchmark for General Information-Seeking Assistant

TL;DR

GISA targets general information-seeking by addressing weaknesses in prior benchmarks through 373 human-designed queries, four structured answer formats, a live subset to resist memorization, and complete human search trajectories for process supervision. It unifies deep reasoning and broad information aggregation within realistic tasks and provides deterministic evaluation via exact-match and supplementary metrics. Experimental results show only modest performance from current agents, underscoring deficiencies in problem decomposition, planning, and efficient tool use, while highlighting the value of live data and trajectory-based supervision for future progress. Overall, GISA offers a robust, dynamic framework to study and improve general-purpose information-seeking agents in real-world web environments.

Abstract

The advancement of large language models (LLMs) has significantly accelerated the development of search agents capable of autonomously gathering information through multi-turn web interactions. Various benchmarks have been proposed to evaluate such agents. However, existing benchmarks often construct queries backward from answers, producing unnatural tasks misaligned with real-world needs. Moreover, these benchmarks tend to focus on either locating specific information or aggregating information from multiple sources, while relying on static answer sets prone to data contamination. To bridge these gaps, we introduce GISA, a benchmark for General Information-Seeking Assistants comprising 373 human-crafted queries that reflect authentic information-seeking scenarios. GISA features four structured answer formats (item, set, list, and table), enabling deterministic evaluation. It integrates both deep reasoning and broad information aggregation within unified tasks, and includes a live subset with periodically updated answers to resist memorization. Notably, GISA provides complete human search trajectories for every query, offering gold-standard references for process-level supervision and imitation learning. Experiments on mainstream LLMs and commercial search products reveal that even the best-performing model achieves only 19.30\% exact match score, with performance notably degrading on tasks requiring complex planning and comprehensive information gathering. These findings highlight substantial room for future improvement.

Figures (10)

• Figure 1: Illustration of benchmark construction process and some example queries with various types of answers.
• Figure 2: Distribution of topics in GISA.
• Figure 3: Statistics of human annotation time on GISA.
• Figure 4: Performance of Kimi K2.5 (thinking) and Claude 4.5 (thinking) on different subsets of GISA.
• Figure 5: Inference-time scaling performance of Qwen3-Max over 40 random samples.