Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol

Jie Zhu, Yimin Tian, Boyang Li, Kehao Wu, Zhongzhi Liang, Junhui Li, Xianyin Zhang, Lifan Guo, Feng Chen, Yong Liu, Chi Zhang

Abstract

This paper introduces \textbf{FinMCP-Bench}, a novel benchmark for evaluating large language models (LLMs) in solving real-world financial problems through tool invocation of financial model context protocols. FinMCP-Bench contains 613 samples spanning 10 main scenarios and 33 sub-scenarios, featuring both real and synthetic user queries to ensure diversity and authenticity. It incorporates 65 real financial MCPs and three types of samples, single tool, multi-tool, and multi-turn, allowing evaluation of models across different levels of task complexity. Using this benchmark, we systematically assess a range of mainstream LLMs and propose metrics that explicitly measure tool invocation accuracy and reasoning capabilities. FinMCP-Bench provides a standardized, practical, and challenging testbed for advancing research on financial LLM agents.

FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol

Abstract

This paper introduces \textbf{FinMCP-Bench}, a novel benchmark for evaluating large language models (LLMs) in solving real-world financial problems through tool invocation of financial model context protocols. FinMCP-Bench contains 613 samples spanning 10 main scenarios and 33 sub-scenarios, featuring both real and synthetic user queries to ensure diversity and authenticity. It incorporates 65 real financial MCPs and three types of samples, single tool, multi-tool, and multi-turn, allowing evaluation of models across different levels of task complexity. Using this benchmark, we systematically assess a range of mainstream LLMs and propose metrics that explicitly measure tool invocation accuracy and reasoning capabilities. FinMCP-Bench provides a standardized, practical, and challenging testbed for advancing research on financial LLM agents.

Paper Structure

This paper contains 13 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. FinMCP-Bench: A Financial MCP Benchmark
  3. Data Source
  4. Chain-based Multi-tool Sample Construction
  5. Role-Playing-based Multi-turn Sample Construction
  6. Quality Control
  7. Dataset Analysis
  8. Experimentation
  9. Experimental Settings
  10. Evaluation Metrics
  11. Experimental Results
  12. Experimental Analysis
  13. Conclusion

Figures (5)

  • Figure 1: The 10 main scenarios and 33 sub-scenarios in FinMCP-Bench.
  • Figure 2: Overview of the chain-based method for synthesizing multi-tool samples.
  • Figure 3: Overview of the role-playing-based method for synthesizing multi-turn samples.
  • Figure 4: Scenario-wise TF1 results on FinMCP-Bench. Full scenario names are listed in Figure \ref{['fig:category']}.
  • Figure 5: Difficulty-wise TF1 on FinMCP-Bench