UniFinEval: Towards Unified Evaluation of Financial Multimodal Models across Text, Images and Videos

TL;DR

UniFinEval introduces the first unified, high-information-density multimodal benchmark for finance, integrating text, images, and videos to evaluate models across five real-world scenarios. Built manually by financial experts, the 3,767 bilingual Q&A dataset targets cross-modal consistency and multi-hop reasoning from perception to expert-level decision-making, with robust quality-control processes. Evaluations of 10 mainstream MLLMs under Zero-Shot and CoT settings show Gemini-3-pro-preview as the best performer but still far behind human experts, especially in cross-modal alignment and complex decision tasks. The work also provides comprehensive error analyses and perturbation designs, offering practical directions to improve robustness and applicability of financial MLLMs in practice, with data and code publicly available. The benchmark aims to guide safer, more reliable deployment of multimodal financial intelligence tools in real-world operations.

Abstract

Multimodal large language models are playing an increasingly significant role in empowering the financial domain, however, the challenges they face, such as multimodal and high-density information and cross-modal multi-hop reasoning, go beyond the evaluation scope of existing multimodal benchmarks. To address this gap, we propose UniFinEval, the first unified multimodal benchmark designed for high-information-density financial environments, covering text, images, and videos. UniFinEval systematically constructs five core financial scenarios grounded in real-world financial systems: Financial Statement Auditing, Company Fundamental Reasoning, Industry Trend Insights, Financial Risk Sensing, and Asset Allocation Analysis. We manually construct a high-quality dataset consisting of 3,767 question-answer pairs in both chinese and english and systematically evaluate 10 mainstream MLLMs under Zero-Shot and CoT settings. Results show that Gemini-3-pro-preview achieves the best overall performance, yet still exhibits a substantial gap compared to financial experts. Further error analysis reveals systematic deficiencies in current models. UniFinEval aims to provide a systematic assessment of MLLMs' capabilities in fine-grained, high-information-density financial environments, thereby enhancing the robustness of MLLMs applications in real-world financial scenarios. Data and code are available at https://github.com/aifinlab/UniFinEval.

Figures (21)

• Figure 1: UniFinEval is manually constructed and supports full-modality inputs including text, images, and videos. It is equipped with cross-modal reasoning capabilities and features high information density while closely aligning with real financial business practices.
• Figure 2: UniFinEval covers five major financial scenarios and constructs datasets spanning text, images, videos, as well as multiple cross-modal combinations. It features high-information-density and manually construct data, together with dedicated designs for cross-modal consistency checking and multi-Hop reasoning, providing comprehensive support for MLLMs evaluation in financial domains.
• Figure 3: An example of a cross-modal multi-hop question in UniFinEval. The answer is derived from the acquisition and integration of key information from the presented text, images, and videos.
• Figure 4: As evident from the visualization of the result comparisons, the performance of the vast majority of models achieved a slight improvement under the CoT evaluation setting, though the overall magnitude of this enhancement remains relatively limited.
• Figure 5: The radar chart summarizes the relative proportions of five major categories of errors observed in incorrect model predictions. Each axis reflects the proportion of a specific error type among all erroneous cases for a given model, highlighting differences in error concentration and reasoning weaknesses across models.