FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning

TL;DR

FinSearchComp introduces the first fully open-source, end-to-end benchmark for realistic financial search and reasoning, spanning 635 expert-curated questions across Global and Greater China and three analyst-style task families. It combines a rigorous, rubric-based evaluation with an LLM-as-a-Judge to assess time-sensitive data fetching, historical lookups, and multi-source historical investigations, guided by a multi-stage quality-control pipeline. The experimental results show web-enabled models with financial plugins approaching expert performance on the global subset but reveal persistent gaps in freshness, provenance, and cross-source reconciliation, with pronounced regional differences. The work also presents the Xpert Platform and Leaderboard as mechanisms to scale expert-level evaluation, underscoring the benchmark’s practical relevance for advancing robust financial agents.

Abstract

Search has emerged as core infrastructure for LLM-based agents and is widely viewed as critical on the path toward more general intelligence. Finance is a particularly demanding proving ground: analysts routinely conduct complex, multi-step searches over time-sensitive, domain-specific data, making it ideal for assessing both search proficiency and knowledge-grounded reasoning. Yet no existing open financial datasets evaluate data searching capability of end-to-end agents, largely because constructing realistic, complicated tasks requires deep financial expertise and time-sensitive data is hard to evaluate. We present FinSearchComp, the first fully open-source agent benchmark for realistic, open-domain financial search and reasoning. FinSearchComp comprises three tasks -- Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation -- closely reproduce real-world financial analyst workflows. To ensure difficulty and reliability, we engage 70 professional financial experts for annotation and implement a rigorous multi-stage quality-assurance pipeline. The benchmark includes 635 questions spanning global and Greater China markets, and we evaluate 21 models (products) on it. Grok 4 (web) tops the global subset, approaching expert-level accuracy. DouBao (web) leads on the Greater China subset. Experimental analyses show that equipping agents with web search and financial plugins substantially improves results on FinSearchComp, and the country origin of models and tools impact performance significantly.By aligning with realistic analyst tasks and providing end-to-end evaluation, FinSearchComp offers a professional, high-difficulty testbed for complex financial search and reasoning.

Figures (9)

• Figure 1: The performance of web-based products on the global subset (left) and the Greater China subset (right) of FinSearchComp. Note that the performance of human experts is $75.0$ and $88.3$ on the Global and Greater China subsets, respectively.
• Figure 2: The overview of the construction process. The construction of this benchmark involves three distinct tasks. The data for each task originate from different sources and undergo separate processing pipelines. A uniform quality control procedure is applied across all tasks.
• Figure 3: Data statistics of FinSearchComp.
• Figure 4: Topic distributions in FinSearchComp.
• Figure 5: The performance of various models across the three tasks on FinSearchComp. Models with $0$ scores are all APIs.