Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Financial VQA in Vision Language Models using Intermediate Structured Representations

Archita Srivastava, Abhas Kumar, Rajesh Kumar, Prabhakar Srinivasan

TL;DR

This study investigates the fine-tuning of DEPLOT, a modality conversion module that translates the image of a plot or chart to a linearized table, on a custom dataset of 50,000 bar charts, demonstrating that providing a structured intermediate table alongside the image significantly enhances LLM reasoning performance compared to direct image queries.

Abstract

Chart interpretation is crucial for visual data analysis, but accurately extracting information from charts poses significant challenges for automated models. This study investigates the fine-tuning of DEPLOT, a modality conversion module that translates the image of a plot or chart to a linearized table, on a custom dataset of 50,000 bar charts. The dataset comprises simple, stacked, and grouped bar charts, targeting the unique structural features of these visualizations. The finetuned DEPLOT model is evaluated against its base version using a test set of 1,000 images and two metrics: Relative Mapping Similarity (RMS), which measures categorical mapping accuracy, and Relative Number Set Similarity (RNSS), which evaluates numerical interpretation accuracy. To further explore the reasoning capabilities of large language models (LLMs), we curate an additional set of 100 bar chart images paired with question answer sets. Our findings demonstrate that providing a structured intermediate table alongside the image significantly enhances LLM reasoning performance compared to direct image queries.

Enhancing Financial VQA in Vision Language Models using Intermediate Structured Representations

TL;DR

This study investigates the fine-tuning of DEPLOT, a modality conversion module that translates the image of a plot or chart to a linearized table, on a custom dataset of 50,000 bar charts, demonstrating that providing a structured intermediate table alongside the image significantly enhances LLM reasoning performance compared to direct image queries.

Abstract

Chart interpretation is crucial for visual data analysis, but accurately extracting information from charts poses significant challenges for automated models. This study investigates the fine-tuning of DEPLOT, a modality conversion module that translates the image of a plot or chart to a linearized table, on a custom dataset of 50,000 bar charts. The dataset comprises simple, stacked, and grouped bar charts, targeting the unique structural features of these visualizations. The finetuned DEPLOT model is evaluated against its base version using a test set of 1,000 images and two metrics: Relative Mapping Similarity (RMS), which measures categorical mapping accuracy, and Relative Number Set Similarity (RNSS), which evaluates numerical interpretation accuracy. To further explore the reasoning capabilities of large language models (LLMs), we curate an additional set of 100 bar chart images paired with question answer sets. Our findings demonstrate that providing a structured intermediate table alongside the image significantly enhances LLM reasoning performance compared to direct image queries.
Paper Structure (27 sections, 18 equations, 10 figures, 7 tables)

This paper contains 27 sections, 18 equations, 10 figures, 7 tables.

Table of Contents

  1. Abstract
  2. 1. Introduction
  3. 2. Problem Statement
  4. 3. Prior Methodologies
  5. 4. Our Proposed Solution
  6. 5. Dataset Creation
  7. 6. Evaluation of Base DEPLOT Model on Custom Dataset
  8. 7. Fine-Tuning DEPLOT on the Custom Bar Chart Dataset
  9. 8. Evaluation of Fine-Tuned DEPLOT Model
  10. 9. Improving Question Answering: A Table-Driven Approach
  11. 7. Conclusion and Future Directions
  12. Acknowledgments

Figures (10)

  • Figure 1: Simple Vertical Bar Chart
  • Figure 2: Stacked Horizontal Bar Chart
  • Figure 3: Grouped Vertical Bar Chart
  • Figure 4: ($G_t)$: "TITLE | Sustainable Development Goals <0x0A> Interest_Expense($) | Product Customization Metric <0x0A> Cash | -16 <0x0A> Audit | -8 <0x0A> Operating | 30 <0x0A> Balances | -3"
  • Figure 5: ($G_t)$: "TITLE | Corporate Ethics <0x0A> Product Reliability Metric | Shimane Bank | Handelsbanken <0x0A> Leverage | 285 | 328 <0x0A> Investments | 191 | 418 <0x0A> Accounting | 173 | 127 <0x0A> Payroll | 209 | 294"
  • ...and 5 more figures