FCMBench: A Comprehensive Financial Credit Multimodal Benchmark for Real-world Applications
Yehui Yang, Dalu Yang, Wenshuo Zhou, Fangxin Shang, Yifan Liu, Jie Ren, Haojun Fei, Qing Yang, Tao Chen
TL;DR
FCMBench addresses the absence of a domain-specific multimodal benchmark for financial credit by introducing FCMBench-V1.0, a privacy-conscious dataset with 8,446 QA samples and 4,043 images. It proposes a Perception-Reasoning-Robustness三-dimensional evaluation framework to mirror real-world credit workflows, implemented through a closed synthesis-capture pipeline to avoid data leakage. The benchmark covers 18 certificate types across 3 task families (perception, reasoning, robustness) and includes a standardized JSON-based evaluation protocol, enabling end-to-end assessment of 23 state-of-the-art vision-language models and an in-house baseline. Results show clear differentiation among models, reveal robustness gaps under acquisition artifacts, and demonstrate practical value for industry-academic collaboration and deployment of reliable credit AI systems.
Abstract
As multimodal AI becomes widely used for credit risk assessment and document review, a domain-specific benchmark is urgently needed that (1) reflects documents and workflows specific to financial credit applications, (2) includes credit-specific understanding and real-world robustness, and (3) preserves privacy compliance without sacrificing practical utility. Here, we introduce FCMBench-V1.0 -- a large-scale financial credit multimodal benchmark for real-world applications, covering 18 core certificate types, with 4,043 privacy-compliant images and 8,446 QA samples. The FCMBench evaluation framework consists of three dimensions: Perception, Reasoning, and Robustness, including 3 foundational perception tasks, 4 credit-specific reasoning tasks that require decision-oriented understanding of visual evidence, and 10 real-world acquisition artifact types for robustness stress testing. To reconcile compliance with realism, we construct all samples via a closed synthesis-capture pipeline: we manually synthesize document templates with virtual content and capture scenario-aware images in-house. This design also mitigates pre-training data leakage by avoiding web-sourced or publicly released images. FCMBench can effectively discriminate performance disparities and robustness across modern vision-language models. Extensive experiments were conducted on 23 state-of-the-art vision-language models (VLMs) from 14 top AI companies and research institutes. Among them, Gemini 3 Pro achieves the best F1(\%) score as a commercial model (64.61), Qwen3-VL-235B achieves the best score as an open-source baseline (57.27), and our financial credit-specific model, Qfin-VL-Instruct, achieves the top overall score (64.92). Robustness evaluations show that even top-performing models suffer noticeable performance drops under acquisition artifacts.