Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation

Chenxi Zhang, Ziliang Gan, Liyun Zhu, Youwei Pang, Qing Zhang, Rongjunchen Zhang

TL;DR

FinMTM addresses the gap in financial visual-language benchmarking by introducing a large, bilingual, multi-turn multimodal dataset with 11,133 QA pairs across 3600 images and 400 PDFs. It provides task-specific evaluation protocols including set-overlap scoring, turn-/session-level scoring, and trajectory-based agent evaluation, enabling holistic assessment of perception, reasoning, and tool-use. Across 22 VLMs, results reveal persistent bottlenecks in fine-grained visual grounding, long-context reasoning, and agent workflows, despite strong single-turn perception. FinMTM thus offers a realistic, scalable platform for developing and benchmarking financial VLMs and guiding future improvements in long-context understanding and tool-enabled decision-making.

Abstract

The financial domain poses substantial challenges for vision-language models (VLMs) due to specialized chart formats and knowledge-intensive reasoning requirements. However, existing financial benchmarks are largely single-turn and rely on a narrow set of question formats, limiting comprehensive evaluation in realistic application scenarios. To address this gap, we propose FinMTM, a multi-turn multimodal benchmark that expands diversity along both data and task dimensions. On the data side, we curate and annotate 11{,}133 bilingual (Chinese and English) financial QA pairs grounded in financial visuals, including candlestick charts, statistical plots, and report figures. On the task side, FinMTM covers single- and multiple-choice questions, multi-turn open-ended dialogues, and agent-based tasks. We further design task-specific evaluation protocols, including a set-overlap scoring rule for multiple-choice questions, a weighted combination of turn-level and session-level scores for multi-turn dialogues, and a composite metric that integrates planning quality with final outcomes for agent tasks. Extensive experimental evaluation of 22 VLMs reveal their limitations in fine-grained visual perception, long-context reasoning, and complex agent workflows.

FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation

TL;DR

FinMTM addresses the gap in financial visual-language benchmarking by introducing a large, bilingual, multi-turn multimodal dataset with 11,133 QA pairs across 3600 images and 400 PDFs. It provides task-specific evaluation protocols including set-overlap scoring, turn-/session-level scoring, and trajectory-based agent evaluation, enabling holistic assessment of perception, reasoning, and tool-use. Across 22 VLMs, results reveal persistent bottlenecks in fine-grained visual grounding, long-context reasoning, and agent workflows, despite strong single-turn perception. FinMTM thus offers a realistic, scalable platform for developing and benchmarking financial VLMs and guiding future improvements in long-context understanding and tool-enabled decision-making.

Abstract

The financial domain poses substantial challenges for vision-language models (VLMs) due to specialized chart formats and knowledge-intensive reasoning requirements. However, existing financial benchmarks are largely single-turn and rely on a narrow set of question formats, limiting comprehensive evaluation in realistic application scenarios. To address this gap, we propose FinMTM, a multi-turn multimodal benchmark that expands diversity along both data and task dimensions. On the data side, we curate and annotate 11{,}133 bilingual (Chinese and English) financial QA pairs grounded in financial visuals, including candlestick charts, statistical plots, and report figures. On the task side, FinMTM covers single- and multiple-choice questions, multi-turn open-ended dialogues, and agent-based tasks. We further design task-specific evaluation protocols, including a set-overlap scoring rule for multiple-choice questions, a weighted combination of turn-level and session-level scores for multi-turn dialogues, and a composite metric that integrates planning quality with final outcomes for agent tasks. Extensive experimental evaluation of 22 VLMs reveal their limitations in fine-grained visual perception, long-context reasoning, and complex agent workflows.
Paper Structure (34 sections, 6 equations, 12 figures, 10 tables)

This paper contains 34 sections, 6 equations, 12 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Financial Multimodal Benchmarks
  4. FinMTM Benchmark
  5. Task Definition
  6. Dataset Construction
  7. Dataset Statistics
  8. Comprehensive Financial Evaluation
  9. Objective Questions Evaluation
  10. Open-Ended Questions Evaluation
  11. Turn-level Evaluation.
  12. Session-level Evaluation.
  13. Overall Score.
  14. Financial Agent Evaluation
  15. Experiment
  16. ...and 19 more sections

Figures (12)

  • Figure 1: The radar chart of the overall performance across different dimensions.
  • Figure 2: Comparison of leading VLMs on general financial capabilities on FinMTM benchmark. The final score is calculated as the average of three task scores: objective questions, open-ended questions, and financial agent.
  • Figure 3: The proposed data synthesis pipeline tailored for financial agent question generation.
  • Figure 4: Overview of the proposed FinMTM.
  • Figure 5: This figure presents the distribution of error types, based on 1133 responses sampled from Gemini 3 Pro. Each response was manually reviewed, and categorized into distinct error types.
  • ...and 7 more figures