Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models

Gagan Bhatia, El Moatez Billah Nagoudi, Hasan Cavusoglu, Muhammad Abdul-Mageed

TL;DR

FinTral presents a family of multimodal financial LLMs built on Mistral-7B, capable of processing textual, numerical, tabular, and visual data. The authors assemble FinSet, a 20B-token finance-focused corpus, and employ instruction tuning, direct preference optimization (DPO), AI feedback, and retrieval/tool augmentation to achieve strong zero-shot and few-shot performance, often surpassing open baselines and approaching GPT-4 on multiple tasks. A dedicated multimodal evaluation suite demonstrates robust chart understanding, hallucination mitigation, and real-time analytical potential, while analyses discuss limitations and ethical considerations. The work offers a scalable blueprint for finance-specific LLMs with strong reasoning, numerical handling, and visual data interpretation, enabled by a combination of data curation, alignment methods, and retrieval-enhanced inference.

Abstract

We introduce FinTral, a suite of state-of-the-art multimodal large language models (LLMs) built upon the Mistral-7b model and tailored for financial analysis. FinTral integrates textual, numerical, tabular, and image data. We enhance FinTral with domain-specific pretraining, instruction fine-tuning, and RLAIF training by exploiting a large collection of textual and visual datasets we curate for this work. We also introduce an extensive benchmark featuring nine tasks and 25 datasets for evaluation, including hallucinations in the financial domain. Our FinTral model trained with direct preference optimization employing advanced Tools and Retrieval methods, dubbed FinTral-DPO-T&R, demonstrates an exceptional zero-shot performance. It outperforms ChatGPT-3.5 in all tasks and surpasses GPT-4 in five out of nine tasks, marking a significant advancement in AI-driven financial technology. We also demonstrate that FinTral has the potential to excel in real-time analysis and decision-making in diverse financial contexts. The GitHub repository for FinTral is available at \url{https://github.com/UBC-NLP/fintral}.

FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models

TL;DR

FinTral presents a family of multimodal financial LLMs built on Mistral-7B, capable of processing textual, numerical, tabular, and visual data. The authors assemble FinSet, a 20B-token finance-focused corpus, and employ instruction tuning, direct preference optimization (DPO), AI feedback, and retrieval/tool augmentation to achieve strong zero-shot and few-shot performance, often surpassing open baselines and approaching GPT-4 on multiple tasks. A dedicated multimodal evaluation suite demonstrates robust chart understanding, hallucination mitigation, and real-time analytical potential, while analyses discuss limitations and ethical considerations. The work offers a scalable blueprint for finance-specific LLMs with strong reasoning, numerical handling, and visual data interpretation, enabled by a combination of data curation, alignment methods, and retrieval-enhanced inference.

Abstract

We introduce FinTral, a suite of state-of-the-art multimodal large language models (LLMs) built upon the Mistral-7b model and tailored for financial analysis. FinTral integrates textual, numerical, tabular, and image data. We enhance FinTral with domain-specific pretraining, instruction fine-tuning, and RLAIF training by exploiting a large collection of textual and visual datasets we curate for this work. We also introduce an extensive benchmark featuring nine tasks and 25 datasets for evaluation, including hallucinations in the financial domain. Our FinTral model trained with direct preference optimization employing advanced Tools and Retrieval methods, dubbed FinTral-DPO-T&R, demonstrates an exceptional zero-shot performance. It outperforms ChatGPT-3.5 in all tasks and surpasses GPT-4 in five out of nine tasks, marking a significant advancement in AI-driven financial technology. We also demonstrate that FinTral has the potential to excel in real-time analysis and decision-making in diverse financial contexts. The GitHub repository for FinTral is available at \url{https://github.com/UBC-NLP/fintral}.
Paper Structure (24 sections, 14 figures, 9 tables)

This paper contains 24 sections, 14 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. FinSet
  4. Pretraining Dataset
  5. Financial Instruction Data
  6. Financial AI Feedback Data
  7. Visual Financial Instruction Dataset
  8. Downstream Evaluation Datasets
  9. Fintral
  10. Experiments
  11. Instruction Tuning and Model Alignment
  12. Retrieval and Tools Usage
  13. Multimodal Evaluation
  14. Financial Hallucination Evaluation
  15. Discussion
  16. ...and 9 more sections

Figures (14)

  • Figure 1: Comparative Performance Analysis on text-based tasks of Key Financial AI Models. We compare three variations of FinTral with ChatGPT (GPT-3.5) and GPT-4 across seven task clusters: Sentiment Analysis (SA), Named Entity Recognition (NER), Number Understanding (NU), Text Summarization (TS), Stock Movement Prediction (SMP), Credit Scoring (CS), and Firm Disclosure (FD).
  • Figure 2: FinSET, a Financial Training and Evaluation Benchmark.
  • Figure 3: Illustration of our data deduplication pipeline
  • Figure 4: Illustration of the FinVQA Dataset to understand model's proficiency in Chart Understanding Tasks. The figure demonstrates the model's ability to interpret stock price movements and identify peak values from a candlestick chart.
  • Figure 5: FinTral prompting method
  • ...and 9 more figures