Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FinMMDocR: Benchmarking Financial Multimodal Reasoning with Scenario Awareness, Document Understanding, and Multi-Step Computation

Zichen Tang, Haihong E, Rongjin Li, Jiacheng Liu, Linwei Jia, Zhuodi Hao, Zhongjun Yang, Yuanze Li, Haolin Tian, Xinyi Hu, Peizhi Zhao, Yuan Liu, Zhengyu Wang, Xianghe Wang, Yiling Huang, Xueyuan Lin, Ruofei Bai, Zijian Xie, Qian Huang, Ruining Cao, Haocheng Gao

TL;DR

FinMMDocR tackles the gap in financial multimodal numerical reasoning by providing a large bilingual benchmark with scenario-aware problems, visually rich long documents, and explicit multi-step computation tasks. The dataset comprises 1,200 questions paired with 837 long financial documents, 12 implicit scenario types, and cross-page evidence requiring an average of 11 reasoning steps. Experiments show no model reaches expert-level accuracy, with the best around 58–60 percent, and reveal distinct roles for document understanding, extraction, and multimodal perception, as well as strengths and limits of retrieval-augmented strategies. The work sets a benchmark for advancing domain-specific multimodal reasoning in finance and motivates methodological improvements in both model architectures and retrieval frameworks to better support real-world decision making.

Abstract

We introduce FinMMDocR, a novel bilingual multimodal benchmark for evaluating multimodal large language models (MLLMs) on real-world financial numerical reasoning. Compared to existing benchmarks, our work delivers three major advancements. (1) Scenario Awareness: 57.9% of 1,200 expert-annotated problems incorporate 12 types of implicit financial scenarios (e.g., Portfolio Management), challenging models to perform expert-level reasoning based on assumptions; (2) Document Understanding: 837 Chinese/English documents spanning 9 types (e.g., Company Research) average 50.8 pages with rich visual elements, significantly surpassing existing benchmarks in both breadth and depth of financial documents; (3) Multi-Step Computation: Problems demand 11-step reasoning on average (5.3 extraction + 5.7 calculation steps), with 65.0% requiring cross-page evidence (2.4 pages average). The best-performing MLLM achieves only 58.0% accuracy, and different retrieval-augmented generation (RAG) methods show significant performance variations on this task. We expect FinMMDocR to drive improvements in MLLMs and reasoning-enhanced methods on complex multimodal reasoning tasks in real-world scenarios.

FinMMDocR: Benchmarking Financial Multimodal Reasoning with Scenario Awareness, Document Understanding, and Multi-Step Computation

TL;DR

FinMMDocR tackles the gap in financial multimodal numerical reasoning by providing a large bilingual benchmark with scenario-aware problems, visually rich long documents, and explicit multi-step computation tasks. The dataset comprises 1,200 questions paired with 837 long financial documents, 12 implicit scenario types, and cross-page evidence requiring an average of 11 reasoning steps. Experiments show no model reaches expert-level accuracy, with the best around 58–60 percent, and reveal distinct roles for document understanding, extraction, and multimodal perception, as well as strengths and limits of retrieval-augmented strategies. The work sets a benchmark for advancing domain-specific multimodal reasoning in finance and motivates methodological improvements in both model architectures and retrieval frameworks to better support real-world decision making.

Abstract

We introduce FinMMDocR, a novel bilingual multimodal benchmark for evaluating multimodal large language models (MLLMs) on real-world financial numerical reasoning. Compared to existing benchmarks, our work delivers three major advancements. (1) Scenario Awareness: 57.9% of 1,200 expert-annotated problems incorporate 12 types of implicit financial scenarios (e.g., Portfolio Management), challenging models to perform expert-level reasoning based on assumptions; (2) Document Understanding: 837 Chinese/English documents spanning 9 types (e.g., Company Research) average 50.8 pages with rich visual elements, significantly surpassing existing benchmarks in both breadth and depth of financial documents; (3) Multi-Step Computation: Problems demand 11-step reasoning on average (5.3 extraction + 5.7 calculation steps), with 65.0% requiring cross-page evidence (2.4 pages average). The best-performing MLLM achieves only 58.0% accuracy, and different retrieval-augmented generation (RAG) methods show significant performance variations on this task. We expect FinMMDocR to drive improvements in MLLMs and reasoning-enhanced methods on complex multimodal reasoning tasks in real-world scenarios.
Paper Structure (19 sections, 6 figures, 3 tables)

This paper contains 19 sections, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Benchmark Construction
  3. Overview of FinMMDocR
  4. Data Curation Process
  5. Updates to Public Dataset
  6. Building a Novel Dataset from Scratch
  7. Data Quality Assurance
  8. Benchmark Analysis
  9. Experiments
  10. Experiments Setting
  11. Models
  12. Input Paradigm
  13. Evaluation Methods
  14. Main Results
  15. Fine-Grained Analysis
  16. ...and 4 more sections

Figures (6)

  • Figure 1: An example of FinMMDocR, including a real-world scenario, a visually-rich document and a multi-step numerical reasoning question, demanding models to reason about China’s import volume shifts for Brazil vs. US soybeans based on evolving US-China tariff conflicts.
  • Figure 2: 12 financial scenarios with FinMMDocR examples, covering 9 document categories and cross-page computations. Requires expert scenario awareness, document understanding, and multi-step computation. Kws: keywords, GT: ground truth.
  • Figure 3: Distribution of FinMMDocR: financial scenarios, document page lengths, and reasoning steps per question.
  • Figure 4: Distribution of FinMMDocR: financial document categories.
  • Figure 5: Fine-grained results based on (top left) scenario count, (bottom left) document length, (bottom middle) average evidence position, and (right) the number of steps in numerical extraction, numerical calculation, and overall computation.
  • ...and 1 more figures