FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding

TL;DR

FlowLearn introduces two complementary flowchart datasets (3,858 scientific and 10,000 simulated Mermaid-generated charts) with rich annotations, enabling rigorous LVLM evaluation across OCR, factual reasoning, description, and Mermaid-code generation. The authors systematically benchmark state-of-the-art LVLMs, revealing that no model excels across all tasks and that challenging areas include precise text extraction, structural reasoning, and code synthesis. Through extensive experiments and ablation studies, FlowLearn highlights specific gaps in current LVLM capabilities for flowchart understanding and provides a scalable benchmark and evaluation protocol to drive future improvements. The work offers actionable insights for developing better visual reasoning systems in scientific contexts and encourages expanding benchmarks to broader diagram types and richer supervision.

Abstract

Flowcharts are graphical tools for representing complex concepts in concise visual representations. This paper introduces the FlowLearn dataset, a resource tailored to enhance the understanding of flowcharts. FlowLearn contains complex scientific flowcharts and simulated flowcharts. The scientific subset contains 3,858 flowcharts sourced from scientific literature and the simulated subset contains 10,000 flowcharts created using a customizable script. The dataset is enriched with annotations for visual components, OCR, Mermaid code representation, and VQA question-answer pairs. Despite the proven capabilities of Large Vision-Language Models (LVLMs) in various visual understanding tasks, their effectiveness in decoding flowcharts - a crucial element of scientific communication - has yet to be thoroughly investigated. The FlowLearn test set is crafted to assess the performance of LVLMs in flowchart comprehension. Our study thoroughly evaluates state-of-the-art LVLMs, identifying existing limitations and establishing a foundation for future enhancements in this relatively underexplored domain. For instance, in tasks involving simulated flowcharts, GPT-4V achieved the highest accuracy (58%) in counting the number of nodes, while Claude recorded the highest accuracy (83%) in OCR tasks. Notably, no single model excels in all tasks within the FlowLearn framework, highlighting significant opportunities for further development.

Figures (1)

• Figure 1: Overview of the FlowLearn Dataset illustrating the detailed components within the Scientific and Simulated subsets.