BizFinBench.v2: A Unified Dual-Mode Bilingual Benchmark for Expert-Level Financial Capability Alignment

TL;DR

BizFinBench.v2 addresses the gap between LLM benchmark performance and real-world financial efficacy by introducing a large-scale benchmark built on authentic cross-market data and including online evaluation. It employs a dual-track framework—Core Business Capabilities plus Online Performance—across four major financial scenarios and eight offline tasks plus two online tasks, totaling 29,578 QA pairs. The study demonstrates that ChatGPT-5 achieves the highest average accuracy (61.5%), with open-source models like Qwen3-235B-A22B-Thinking-2507 close behind (53.3%), while online tasks reveal strong performance from DeepSeek-R1 in asset allocation. Five business-oriented error categories are identified through expert-informed analysis, providing actionable directions for improving LLMs in practical financial contexts and paving the way for deployment-ready, real-time AI in finance.

Abstract

Large language models have undergone rapid evolution, emerging as a pivotal technology for intelligence in financial operations. However, existing benchmarks are often constrained by pitfalls such as reliance on simulated or general-purpose samples and a focus on singular, offline static scenarios. Consequently, they fail to align with the requirements for authenticity and real-time responsiveness in financial services, leading to a significant discrepancy between benchmark performance and actual operational efficacy. To address this, we introduce BizFinBench.v2, the first large-scale evaluation benchmark grounded in authentic business data from both Chinese and U.S. equity markets, integrating online assessment. We performed clustering analysis on authentic user queries from financial platforms, resulting in eight fundamental tasks and two online tasks across four core business scenarios, totaling 29,578 expert-level Q&A pairs. Experimental results demonstrate that ChatGPT-5 achieves a prominent 61.5% accuracy in main tasks, though a substantial gap relative to financial experts persists; in online tasks, DeepSeek-R1 outperforms all other commercial LLMs. Error analysis further identifies the specific capability deficiencies of existing models within practical financial business contexts. BizFinBench.v2 transcends the limitations of current benchmarks, achieving a business-level deconstruction of LLM financial capabilities and providing a precise basis for evaluating efficacy in the widespread deployment of LLMs within the financial domain. The data and code are available at https://github.com/HiThink-Research/BizFinBench.v2.

Figures (22)

• Figure 1: BizFinBench.v2 comprises eight foundational tasks and two online tasks distributed across four major scenarios. The top-right corner displays a real-time screenshot of the Portfolio Asset Allocation task.
• Figure 2: We have ranked the performance of the LLMs participating in the evaluation under the zero-shot setting, and these results reflect their authentic practical business capabilities.
• Figure 3: The platform’s user structure is primarily analyzed through user type, device distribution, and national distribution; here, Inst. serves as an abbreviation for institutional users.
• Figure 4: We selected representative LLMs for error analysis and summarized five typical dilemmas faced by LLMs in actual business scenarios
• Figure 5: An Example of Abnormal Information Tracing. This task requires the LLM to identify relevant information from various given heterogeneous sources that caused the stock price fluctuation of the corresponding company and to provide the associated information index. It evaluates the LLM's information analysis and summarization capabilities. Due to the extensive length of the data, we have truncated the content here and primarily present a sample of the input data.