Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts

Shubhankar Singh, Purvi Chaurasia, Yerram Varun, Pranshu Pandya, Vatsal Gupta, Vivek Gupta, Dan Roth

TL;DR

FlowVQA introduces a challenging flowchart-based multimodal VQA benchmark that probes visual grounding and spatial/logical reasoning in vision-language models. It couples 2,272 flowchart images with 22,413 QA pairs generated through a two-step text-to-flowchart pipeline and validated by human annotators, enabling diverse reasoning tasks from fact retrieval to topological analysis. Comprehensive evaluations across proprietary and open-source VLMs reveal significant gaps in current models, with performance declining as flowchart complexity grows and a notable directional bias when flowcharts are inverted. The work motivates future research in flowchart-aware models, graph-encoder architectures, and counterfactual probing to advance multimodal reasoning capabilities.

Abstract

Existing benchmarks for visual question answering lack in visual grounding and complexity, particularly in evaluating spatial reasoning skills. We introduce FlowVQA, a novel benchmark aimed at assessing the capabilities of visual question-answering multimodal language models in reasoning with flowcharts as visual contexts. FlowVQA comprises 2,272 carefully generated and human-verified flowchart images from three distinct content sources, along with 22,413 diverse question-answer pairs, to test a spectrum of reasoning tasks, including information localization, decision-making, and logical progression. We conduct a thorough baseline evaluation on a suite of both open-source and proprietary multimodal language models using various strategies, followed by an analysis of directional bias. The results underscore the benchmark's potential as a vital tool for advancing the field of multimodal modeling, providing a focused and challenging environment for enhancing model performance in visual and logical reasoning tasks.

FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts

TL;DR

FlowVQA introduces a challenging flowchart-based multimodal VQA benchmark that probes visual grounding and spatial/logical reasoning in vision-language models. It couples 2,272 flowchart images with 22,413 QA pairs generated through a two-step text-to-flowchart pipeline and validated by human annotators, enabling diverse reasoning tasks from fact retrieval to topological analysis. Comprehensive evaluations across proprietary and open-source VLMs reveal significant gaps in current models, with performance declining as flowchart complexity grows and a notable directional bias when flowcharts are inverted. The work motivates future research in flowchart-aware models, graph-encoder architectures, and counterfactual probing to advance multimodal reasoning capabilities.

Abstract

Existing benchmarks for visual question answering lack in visual grounding and complexity, particularly in evaluating spatial reasoning skills. We introduce FlowVQA, a novel benchmark aimed at assessing the capabilities of visual question-answering multimodal language models in reasoning with flowcharts as visual contexts. FlowVQA comprises 2,272 carefully generated and human-verified flowchart images from three distinct content sources, along with 22,413 diverse question-answer pairs, to test a spectrum of reasoning tasks, including information localization, decision-making, and logical progression. We conduct a thorough baseline evaluation on a suite of both open-source and proprietary multimodal language models using various strategies, followed by an analysis of directional bias. The results underscore the benchmark's potential as a vital tool for advancing the field of multimodal modeling, providing a focused and challenging environment for enhancing model performance in visual and logical reasoning tasks.
Paper Structure (21 sections, 7 figures, 9 tables)

This paper contains 21 sections, 7 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Proposed FlowVQA Resource
  3. Flowchart Sources
  4. Flowchart Generation
  5. Question Answer Creation
  6. Human Verification Pipeline and Platform
  7. Experimental Evaluation
  8. Baseline Evaluation
  9. Evaluation Method
  10. Baseline Results and Discussion
  11. Directional Bias
  12. Related Work
  13. Conclusion and Future Work
  14. Flowchart QA Example
  15. Dataset Distribution
  16. ...and 6 more sections

Figures (7)

  • Figure 1: A zoomed-in section of a flowchart in our resource set along with a correspnding QA pair. wiki00203: "How To Convert an Old Google Spreadsheet to Google Sheets." A detailed example of a flowchart along with its question-answer pairs is outlined in Appendix \ref{['apex:flowchart-qa-example']}.
  • Figure 2: Our dataset's generation pipeline encompasses the creation of flowcharts. As previously outlined, we employ a comprehensive two-step process to derive high-quality flowcharts from source texts. Additionally, to guarantee accurate generation, a cross-verification mechanism is implemented.
  • Figure 3: Our dataset incorporates a question creation pipeline tailored to accommodate various question types. As previously noted, each question type undergoes generation via a carefully crafted prompt, meticulously designed to achieve the specific objectives associated with that type of question
  • Figure 4: The figure shows the distribution of our data across different sources as well as across different types of questions.
  • Figure 5: The horizontal bar chart shows the performance of FlowVQA dataset on various modelling strategies outlined in Section \ref{['sec:experiment_evaluation']}.
  • ...and 2 more figures