Overcoming Vision Language Model Challenges in Diagram Understanding: A Proof-of-Concept with XML-Driven Large Language Models Solutions

TL;DR

The paper tackles diagram understanding by bypassing visual recognition in Vision-Language Models and instead extracting diagram metadata from editable source files, such as Excel .xlsx, to feed textual representations to Large Language Models. A proof-of-concept using system design diagrams demonstrates that an XML-driven, text-based approach can yield higher accuracy for structure-aware questions than image-based VLM methods. The authors provide an open-source library to parse Excel XML and show potential generalization to pptx/docx formats, highlighting practical benefits for business workflows. While promising, the work remains preliminary and motivates broader validation, integration with retrieval-augmented generation, and possible hybrid approaches that leverage both text and visuals.

Abstract

Diagrams play a crucial role in visually conveying complex relationships and processes within business documentation. Despite recent advances in Vision-Language Models (VLMs) for various image understanding tasks, accurately identifying and extracting the structures and relationships depicted in diagrams continues to pose significant challenges. This study addresses these challenges by proposing a text-driven approach that bypasses reliance on VLMs' visual recognition capabilities. Instead, it utilizes the editable source files--such as xlsx, pptx or docx--where diagram elements (e.g., shapes, lines, annotations) are preserved as textual metadata. In our proof-of-concept, we extracted diagram information from xlsx-based system design documents and transformed the extracted shape data into textual input for Large Language Models (LLMs). This approach allowed the LLM to analyze relationships and generate responses to business-oriented questions without the bottleneck of image-based processing. Experimental comparisons with a VLM-based method demonstrated that the proposed text-driven framework yielded more accurate answers for questions requiring detailed comprehension of diagram structures.The results obtained in this study are not limited to the tested .xlsx files but can also be extended to diagrams in other documents with source files, such as Office pptx and docx formats. These findings highlight the feasibility of circumventing VLM constraints through direct textual extraction from original source files. By enabling robust diagram understanding through LLMs, our method offers a promising path toward enhanced workflow efficiency and information analysis in real-world business scenarios.

Figures (7)

• Figure 1: Our approach does not rely on the visual recognition of the VLM, but rather inputs and analyzes the LLM as text, with the underlying graphic information stored in the source files referenced to render the Diagram image.
• Figure 2: The diagram used in this study, a system blueprint drawn in Excel using rectangles, text boxes, straight connectors, and bent connectors.
• Figure 3: On the left is an XML excerpt of shape data from an xlsx file, difficult to interpret due to raw numerical values and aliased strings. It also contains excessive, scattered information across multiple files. On the right is a JSON format that parses, transforms, and summarizes meaningful information on connectors and shapes, optimized for diagram understanding and LLM input.
• Figure 4: Comparison of the output of the proposed method (Text-driven) and VLM (Image Input) for understanding entities in a diagram. The figure above shows the input system design diagram. The ID assigned to each shape indicates the ID assigned by the proposed method when passing graphic information by text. Entity comprehension is correct for both methods without omission, and the results show that the XML-driven approach is able to understand components consisting of text boxes and round-rectangles without any group structure information.
• Figure 5: Output comparison of the proposed method (XML-driven) and VLM (Image Input) for understanding relationships in a diagram. The figure above shows the ID assigned to each connector. The connector understanding is correct without omission in the XML-driven approach, while in the Image Input, yellow: probably did not detect the connector bend, red: misidentified a relationship that does not exist, and detected a connector with ID:7 where one of the objects is a text box. The image input is yellow: probably not detecting a connector bend.