Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

Jimin Huang, Mengxi Xiao, Dong Li, Zihao Jiang, Yuzhe Yang, Yifei Zhang, Lingfei Qian, Yan Wang, Xueqing Peng, Yang Ren, Ruoyu Xiang, Zhengyu Chen, Xiao Zhang, Yueru He, Weiguang Han, Shunian Chen, Lihang Shen, Daniel Kim, Yangyang Yu, Yupeng Cao, Zhiyang Deng, Haohang Li, Duanyu Feng, Yongfu Dai, VijayaSai Somasundaram, Peng Lu, Guojun Xiong, Zhiwei Liu, Zheheng Luo, Zhiyuan Yao, Ruey-Ling Weng, Meikang Qiu, Kaleb E Smith, Honghai Yu, Yanzhao Lai, Min Peng, Jian-Yun Nie, Jordan W. Suchow, Xiao-Yang Liu, Benyou Wang, Alejandro Lopez-Lira, Qianqian Xie, Sophia Ananiadou, Junichi Tsujii

TL;DR

Open-FinLLMs address the scarcity of open, multimodal financial AI by introducing FinLLaMA (foundational continual pretraining on 52B tokens across text, tabular, and time-series data), FinLLaMA-Instruct (573K financial instructions), and FinLLaVA (1.43M multimodal tuning pairs). The authors demonstrate through comprehensive zero-shot, few-shot, and supervised evaluations across 14 financial tasks and 30 datasets, plus four multimodal benchmarks, that their open models outperform existing financial LLMs and even rival or surpass GPT-4 family on several finance-specific tasks. The paper contributes an end-to-end training pipeline, multi-modal data curation, and open-source release to accelerate financial AI research and real-world deployment. This work highlights the potential of integrated textual, tabular, time-series, and chart understanding for financial decision-making.

Abstract

Financial LLMs hold promise for advancing financial tasks and domain-specific applications. However, they are limited by scarce corpora, weak multimodal capabilities, and narrow evaluations, making them less suited for real-world application. To address this, we introduce \textit{Open-FinLLMs}, the first open-source multimodal financial LLMs designed to handle diverse tasks across text, tabular, time-series, and chart data, excelling in zero-shot, few-shot, and fine-tuning settings. The suite includes FinLLaMA, pre-trained on a comprehensive 52-billion-token corpus; FinLLaMA-Instruct, fine-tuned with 573K financial instructions; and FinLLaVA, enhanced with 1.43M multimodal tuning pairs for strong cross-modal reasoning. We comprehensively evaluate Open-FinLLMs across 14 financial tasks, 30 datasets, and 4 multimodal tasks in zero-shot, few-shot, and supervised fine-tuning settings, introducing two new multimodal evaluation datasets. Our results show that Open-FinLLMs outperforms afvanced financial and general LLMs such as GPT-4, across financial NLP, decision-making, and multi-modal tasks, highlighting their potential to tackle real-world challenges. To foster innovation and collaboration across academia and industry, we release all codes (https://anonymous.4open.science/r/PIXIU2-0D70/B1D7/LICENSE) and models under OSI-approved licenses.

Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

TL;DR

Open-FinLLMs address the scarcity of open, multimodal financial AI by introducing FinLLaMA (foundational continual pretraining on 52B tokens across text, tabular, and time-series data), FinLLaMA-Instruct (573K financial instructions), and FinLLaVA (1.43M multimodal tuning pairs). The authors demonstrate through comprehensive zero-shot, few-shot, and supervised evaluations across 14 financial tasks and 30 datasets, plus four multimodal benchmarks, that their open models outperform existing financial LLMs and even rival or surpass GPT-4 family on several finance-specific tasks. The paper contributes an end-to-end training pipeline, multi-modal data curation, and open-source release to accelerate financial AI research and real-world deployment. This work highlights the potential of integrated textual, tabular, time-series, and chart understanding for financial decision-making.

Abstract

Financial LLMs hold promise for advancing financial tasks and domain-specific applications. However, they are limited by scarce corpora, weak multimodal capabilities, and narrow evaluations, making them less suited for real-world application. To address this, we introduce \textit{Open-FinLLMs}, the first open-source multimodal financial LLMs designed to handle diverse tasks across text, tabular, time-series, and chart data, excelling in zero-shot, few-shot, and fine-tuning settings. The suite includes FinLLaMA, pre-trained on a comprehensive 52-billion-token corpus; FinLLaMA-Instruct, fine-tuned with 573K financial instructions; and FinLLaVA, enhanced with 1.43M multimodal tuning pairs for strong cross-modal reasoning. We comprehensively evaluate Open-FinLLMs across 14 financial tasks, 30 datasets, and 4 multimodal tasks in zero-shot, few-shot, and supervised fine-tuning settings, introducing two new multimodal evaluation datasets. Our results show that Open-FinLLMs outperforms afvanced financial and general LLMs such as GPT-4, across financial NLP, decision-making, and multi-modal tasks, highlighting their potential to tackle real-world challenges. To foster innovation and collaboration across academia and industry, we release all codes (https://anonymous.4open.science/r/PIXIU2-0D70/B1D7/LICENSE) and models under OSI-approved licenses.
Paper Structure (40 sections, 13 figures, 14 tables)

This paper contains 40 sections, 13 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Financial Large Language Models
  4. Domain Specialization of Large Language Models
  5. Open-FinLLMs: Open Multimodal Financial LLMs
  6. FinLLaMA: Specializing LLaMA3 for Finance with Continual Pre-training
  7. FinLLaMA-Instruct: Domain Task Optimization through Instruction Tuning
  8. FinLLaVA: Enabling Multimodal Capabilities via Multimodal Instruction Tuning
  9. Experiments
  10. Performance of FinLLaMA
  11. Performance of FinLLaMA-Instruct
  12. Performance of FinLLaVA
  13. Conclusion
  14. Curation of Continual Pre-training Corpus
  15. Dataset Details
  16. ...and 25 more sections

Figures (13)

  • Figure 1: Overview of Open-FinLLMs.
  • Figure 2: Example table image. Description: Q: What financial metrics are represented in the table, and how are they organized? A: The table presents various financial metrics organized into rows and columns, with headers indicating the type of data. The first column lists categories such as "Corporate", "For the year ended December 31", and "General and administrative expenses". The corresponding numerical values in the adjacent columns include 628,856, 518,246, 222,216, and 793,595, respectively. The data appears to be structured in a way that highlights different financial activities and their associated values, with totals and segment information also included. For instance, the "Total segment" row shows a value of 886,238, while "Cash and cash equivalents at beginning of period" has a value of 946,929. The table also indicates that the data is presented in millions of dollars, emphasizing the scale of the financial figures.Instruction (Make Comparisons): Q: Which segment had the highest "Total net revenues" and what was the amount? A: The segment with the highest "Total net revenues" is "Millions of dollars" with an amount of 972,582.
  • Figure 3: Example chart image 1. Description: Q: Describe the type of information that the chart provides. A: The chart shows year-over-year growth rates for various countries in 2014 and 2015. Each country has two bars representing growth rates for 2014 (in dark blue) and 2015 (in light blue). The growth rates are measured in percentages and are displayed on the horizontal axis, which ranges from -7.5% to 20%. The countries listed on the left axis include China, Russia, Germany, India, South Korea, United States, Brazil, France, Australia, Canada, United Kingdom, Spain, Netherlands, Italy, and Japan. Key information includes: China's growth rates were 17.8% in 2014 and 16.5% in 2015. Russia had a negative growth rate of -5.4% in 2014 and a positive growth rate of 13.6% in 2015. Germany's growth rates were 7% in 2014 and 10.6% in 2015. India's growth rates were 2.1% in 2014 and 7.6% in 2015. South Korea had growth rates of 5.9% in 2014 and 5.8% in 2015. The United States had growth rates of 5.9% in 2014 and 6.8% in 2015. Brazil had growth rates of 5.9% in 2014 and 12% in 2015.
  • Figure 4: Example chart image 2. Instruction (Make Comparisons): Q: How has the unemployment rate in the given chart changed from 1999 to 2020? A: The unemployment rate in the chart shows significant fluctuations between 1999 and 2020. Starting at 9.64% in 1999, it slightly increased to 9.65% in 2000. There was a sharp decline to 6.81% in 2003 and further to 4.83% in 2005. The rate then fluctuated, peaking at 5.46% in 2009 before decreasing to 4.09% in 2011. It rose again to 5.26% in 2016 and slightly varied around 4.62% in 2018. By 2020, the rate was at 4.81%. Overall, the chart indicates a general downward trend in the unemployment rate over the two decades with occasional increases.
  • Figure 5: Zero-shot performance of FinLLaMA and baseline models.
  • ...and 8 more figures