Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Arrow-Guided VLM: Enhancing Flowchart Understanding via Arrow Direction Encoding

Takamitsu Omasa, Ryo Koshihara, Masumi Morishige

TL;DR

This work addresses the challenge of flowchart reasoning by vision–language models, where topology and arrow directions are often misinterpreted. It proposes a detector–VL M pipeline that preserves geometry through an arrow-aware object detector, OCR-backed text extraction, and graph-structured prompts fed to GPT-4o. On a 90-question benchmark derived from 30 flowcharts, the approach yields about 89% overall accuracy, with a clear 100% accuracy on next-step queries and robust gains without task-specific fine-tuning. Limitations include dependence on detector/OCR precision and a small evaluation set; future work will expand benchmarks and explore BPMN/UML domains.

Abstract

Flowcharts are indispensable tools in software design and business-process analysis, yet current vision-language models (VLMs) frequently misinterpret the directional arrows and graph topology that set these diagrams apart from natural images. We introduce a seven-stage pipeline grouped into three broader processes: (1) arrow-aware detection of nodes and arrow endpoints; (2) optical character recognition (OCR) to extract node text; and (3) construction of a structured prompt that guides the VLMs. Tested on a 90-question benchmark distilled from 30 annotated flowcharts, the method raises overall accuracy from 80 % to 89 % (+9 percentage points) without any task-specific fine-tuning. The gain is most pronounced for next-step queries (25/30 -> 30/30; 100 %, +17 pp); branch-result questions improve more modestly, and before-step questions remain difficult. A parallel evaluation with an LLM-as-a-Judge protocol shows the same trends, reinforcing the advantage of explicit arrow encoding. Limitations include dependence on detector and OCR precision, the small evaluation set, and residual errors at nodes with multiple incoming edges. Future work will enlarge the benchmark with synthetic and handwritten flowcharts and assess the approach on Business Process Model and Notation (BPMN) and Unified Modeling Language (UML).

Arrow-Guided VLM: Enhancing Flowchart Understanding via Arrow Direction Encoding

TL;DR

This work addresses the challenge of flowchart reasoning by vision–language models, where topology and arrow directions are often misinterpreted. It proposes a detector–VL M pipeline that preserves geometry through an arrow-aware object detector, OCR-backed text extraction, and graph-structured prompts fed to GPT-4o. On a 90-question benchmark derived from 30 flowcharts, the approach yields about 89% overall accuracy, with a clear 100% accuracy on next-step queries and robust gains without task-specific fine-tuning. Limitations include dependence on detector/OCR precision and a small evaluation set; future work will expand benchmarks and explore BPMN/UML domains.

Abstract

Flowcharts are indispensable tools in software design and business-process analysis, yet current vision-language models (VLMs) frequently misinterpret the directional arrows and graph topology that set these diagrams apart from natural images. We introduce a seven-stage pipeline grouped into three broader processes: (1) arrow-aware detection of nodes and arrow endpoints; (2) optical character recognition (OCR) to extract node text; and (3) construction of a structured prompt that guides the VLMs. Tested on a 90-question benchmark distilled from 30 annotated flowcharts, the method raises overall accuracy from 80 % to 89 % (+9 percentage points) without any task-specific fine-tuning. The gain is most pronounced for next-step queries (25/30 -> 30/30; 100 %, +17 pp); branch-result questions improve more modestly, and before-step questions remain difficult. A parallel evaluation with an LLM-as-a-Judge protocol shows the same trends, reinforcing the advantage of explicit arrow encoding. Limitations include dependence on detector and OCR precision, the small evaluation set, and residual errors at nodes with multiple incoming edges. Future work will enlarge the benchmark with synthetic and handwritten flowcharts and assess the approach on Business Process Model and Notation (BPMN) and Unified Modeling Language (UML).
Paper Structure (24 sections, 1 figure, 7 tables)

This paper contains 24 sections, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Limitations of End-to-End VLMs on Diagram Tasks
  4. Object-Detection–Driven Flowchart Interpretation
  5. Methodology
  6. Text Extraction via ocr
  7. Object Detection
  8. Integration of Text and Object Information
  9. Arrow Association
  10. Linking Objects and Arrows
  11. Prompt Construction
  12. Question Generation and vlm Inference
  13. Results
  14. Effectiveness of ocr and Detection Model in FlowchartQA
  15. Experimental Setup
  16. ...and 9 more sections

Figures (1)

  • Figure 1: Overview of the seven-stage pipeline: OCR, object detection, text–object fusion, arrow anchoring, node–arrow linking, graph-structured prompt generation, and VLM-based reasoning.