Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agents

Rui Sun, Zuo Bai, Wentao Zhang, Yuxiang Zhang, Li Zhao, Shan Sun, Zhengwen Qiu

TL;DR

FinResearchBench introduces a finance-centric evaluation framework that uses a logic-tree based Agent-as-a-Judge to automatically assess financial research agents. It extracts a logic tree from research outputs and uses this intermediate representation to drive robust, automatic evaluation across 7 task types with 70 finance questions. The framework defines rule-based metrics for analysis width, depth, information density, and paragraph richness, complemented by LLM-based scoring of articulation and professionalism. Experiments show that while top AI agents like Gemini approach human expert quality in several dimensions, human-written reports still outperform AI in holistic assessment, underscoring the value of intermediate reasoning structures and the need for further integration of strengths and improved extraction methods.

Abstract

Recently, AI agents are rapidly evolving in intelligence and widely used in professional research applications, such as STEM, software development, and finance. Among these AI agents, deep research agent is a key category as it can perform long-horizon tasks and solve problems of greater complexity. However, there are few evaluation frameworks and benchmarks that systematically and automatically investigate the capabilities of these research agents. In addition, financial research problems have distinct complexity and subtlety. To fill in the gap, we propose FinResearchBench, which is a logic tree-based Agent-as-a-Judge and targets specifically for the financial research agents. It provides a comprehensive and automatic assessment of the research agents across 7 key types of tasks in the financial research domain. The contributions of this work are two-folded: (1) the first and innovative Agent-as-a-Judge system that extracts the logic tree of the research outcome and uses it as the intermediate information to present a comprehensive, reliable, and robust evaluation; (2) finance-oriented that it covers 70 typical financial research questions, spreading across 7 frequently encountered types of task in the domain.

FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agents

TL;DR

FinResearchBench introduces a finance-centric evaluation framework that uses a logic-tree based Agent-as-a-Judge to automatically assess financial research agents. It extracts a logic tree from research outputs and uses this intermediate representation to drive robust, automatic evaluation across 7 task types with 70 finance questions. The framework defines rule-based metrics for analysis width, depth, information density, and paragraph richness, complemented by LLM-based scoring of articulation and professionalism. Experiments show that while top AI agents like Gemini approach human expert quality in several dimensions, human-written reports still outperform AI in holistic assessment, underscoring the value of intermediate reasoning structures and the need for further integration of strengths and improved extraction methods.

Abstract

Recently, AI agents are rapidly evolving in intelligence and widely used in professional research applications, such as STEM, software development, and finance. Among these AI agents, deep research agent is a key category as it can perform long-horizon tasks and solve problems of greater complexity. However, there are few evaluation frameworks and benchmarks that systematically and automatically investigate the capabilities of these research agents. In addition, financial research problems have distinct complexity and subtlety. To fill in the gap, we propose FinResearchBench, which is a logic tree-based Agent-as-a-Judge and targets specifically for the financial research agents. It provides a comprehensive and automatic assessment of the research agents across 7 key types of tasks in the financial research domain. The contributions of this work are two-folded: (1) the first and innovative Agent-as-a-Judge system that extracts the logic tree of the research outcome and uses it as the intermediate information to present a comprehensive, reliable, and robust evaluation; (2) finance-oriented that it covers 70 typical financial research questions, spreading across 7 frequently encountered types of task in the domain.

Paper Structure

This paper contains 32 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM and Deep Research Agent
  4. General and Domain-Specific Evaluation of Deep Research Agent
  5. FinResearchBench
  6. The Design of FinResearchBench: an Agentic Evaluation System
  7. Logic Tree
  8. Writing Method
  9. Logic Tree Extraction
  10. Logic Tree Based Agent-as-a-Judge
  11. Data Construction
  12. Experiments and Analysis
  13. Experiment Settings
  14. Core Model
  15. Agents and Experiments
  16. ...and 17 more sections

Figures (2)

  • Figure 1: The Illustration of Writing Process
  • Figure 2: The Illustration of Logic Tree Based Agent-as-a-Judge