Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Comprehension to Reasoning: A Hierarchical Benchmark for Automated Financial Research Reporting

Yiyun Zhu, Yidong Jiang, Ziwen Xu, Yinsheng Yao, Dawei Cheng, Jinru Ding, Yejie Zheng, Jie Xu

Abstract

Large language models (LLMs) are increasingly used to generate financial research reports, shifting from auxiliary analytic tools to primary content producers. Yet recent real-world deployments reveal persistent failures--factual errors, numerical inconsistencies, fabricated references, and shallow analysis--that can distort assessments of corporate fundamentals and ultimately trigger severe economic losses. However, existing financial benchmarks focus on comprehension over completed reports rather than evaluating whether a model can produce reliable analysis. Moreover, current evaluation frameworks merely flag hallucinations and lack structured measures for deeper analytical skills, leaving key analytical bottlenecks undiscovered. To address these gaps, we introduce FinReasoning, a benchmark that decomposes Chinese research-report generation into three stages aligned with real analyst workflows, assessing semantic consistency, data alignment, and deep insight. We further propose a fine-grained evaluation framework that strengthens hallucination-correction assessment and incorporates a 12-indicator rubric for core analytical skills. Based on the evaluation results, FinReasoning reveals that most models exhibit a understanding-execution gap: they can identify errors but struggle to generate accurate corrections; they can retrieve data but have difficulty returning it in correct format. Furthermore, no model achieves overwhelming superiority across all three tracks; Doubao-Seed-1.8, GPT-5, and Kimi-K2 rank as the top three in overall performance, yet each exhibits a distinct capability distribution. The evaluation resource is available at https://github.com/TongjiFinLab/FinReasoning.

From Comprehension to Reasoning: A Hierarchical Benchmark for Automated Financial Research Reporting

Abstract

Large language models (LLMs) are increasingly used to generate financial research reports, shifting from auxiliary analytic tools to primary content producers. Yet recent real-world deployments reveal persistent failures--factual errors, numerical inconsistencies, fabricated references, and shallow analysis--that can distort assessments of corporate fundamentals and ultimately trigger severe economic losses. However, existing financial benchmarks focus on comprehension over completed reports rather than evaluating whether a model can produce reliable analysis. Moreover, current evaluation frameworks merely flag hallucinations and lack structured measures for deeper analytical skills, leaving key analytical bottlenecks undiscovered. To address these gaps, we introduce FinReasoning, a benchmark that decomposes Chinese research-report generation into three stages aligned with real analyst workflows, assessing semantic consistency, data alignment, and deep insight. We further propose a fine-grained evaluation framework that strengthens hallucination-correction assessment and incorporates a 12-indicator rubric for core analytical skills. Based on the evaluation results, FinReasoning reveals that most models exhibit a understanding-execution gap: they can identify errors but struggle to generate accurate corrections; they can retrieve data but have difficulty returning it in correct format. Furthermore, no model achieves overwhelming superiority across all three tracks; Doubao-Seed-1.8, GPT-5, and Kimi-K2 rank as the top three in overall performance, yet each exhibits a distinct capability distribution. The evaluation resource is available at https://github.com/TongjiFinLab/FinReasoning.
Paper Structure (21 sections, 4 figures, 5 tables)

This paper contains 21 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning Benchmarks in General Domain
  4. Financial Benchmarks
  5. FinReasoning
  6. Data Source and Statistics
  7. Task Definition and Construction
  8. Semantic Consistency
  9. Data Alignment
  10. Deep Insight
  11. Benchmark Evaluation
  12. Experiments
  13. Experimental Setup
  14. Main Results
  15. Semantic Consistency
  16. ...and 6 more sections

Figures (4)

  • Figure 1: An overview of the taxonomy of FinReasoning benchmark, which is organized into three complementary tracks: Semantic Consistency, Data Alignment, Deep Insight. Each track is constructed with multi-level tasks.
  • Figure 2: An overview of the FinReasoning benchmark, presenting its data sources, the construction procedures for each task, and the data statistics including the number of sub-tasks within each track.
  • Figure 3: Comparative performance across model categories over three level in Data Alignment track.
  • Figure 4: Visualized DI scores across tasks and model series.