Seeing the Forest and the Trees: Solving Visual Graph and Tree Based Data Structure Problems using Large Multimodal Models

TL;DR

This work examines whether large multimodal models can solve graph and tree problems when presented solely as images, addressing concerns about assessment integrity in computing education. It introduces a 9{,}072-task vision-language benchmark and an open-source generator to probe structural and visual variations, evaluating models such as GPT-4o, GPT-4V, Gemini, and Claude. Results show trees are generally easier for LMMs (GPT-4o reaching up to $87.6\%$ pass@3 on trees) while graphs are more challenging (Gemini 1.5 Flash up to $56.2\%$ pass@3), with structural features like edge counts and degree distributions driving accuracy more than cosmetic features. The findings prompt a rethinking of assessment strategies and highlight opportunities for pedagogy, including AI-assisted tutoring and scaffolded learning tools, supported by an openly available benchmark and generator for future research.

Abstract

Recent advancements in generative AI systems have raised concerns about academic integrity among educators. Beyond excelling at solving programming problems and text-based multiple-choice questions, recent research has also found that large multimodal models (LMMs) can solve Parsons problems based only on an image. However, such problems are still inherently text-based and rely on the capabilities of the models to convert the images of code blocks to their corresponding text. In this paper, we further investigate the capabilities of LMMs to solve graph and tree data structure problems based only on images. To achieve this, we computationally construct and evaluate a novel benchmark dataset comprising 9,072 samples of diverse graph and tree data structure tasks to assess the performance of the GPT-4o, GPT-4v, Gemini 1.5 Pro, Gemini 1.5 Flash, Gemini 1.0 Pro Vision, and Claude 3 model families. GPT-4o and Gemini 1.5 Flash performed best on trees and graphs respectively. GPT-4o achieved 87.6% accuracy on tree samples, while Gemini 1.5 Flash, achieved 56.2% accuracy on graph samples. Our findings highlight the influence of structural and visual variations on model performance. This research not only introduces an LMM benchmark to facilitate replication and further exploration but also underscores the potential of LMMs in solving complex computing problems, with important implications for pedagogy and assessment practices.

Figures (6)

• Figure 1: Overview of our benchmark dataset creation process, illustrating the transition from dataset construction to model evaluation. The image prompt set $I$ and text prompt set $Q$ are generated as follows: each image $I_i$ has a combination of attributes where $s$ denotes data structure type, $n$ denotes the number of nodes, $v$ denotes variation of node values, $e$ denotes edge width, and $c$ denotes node color. A text prompt $Q_i$ combines attributes, where $s$ denotes data structure type and $t$ denotes the operational task category. The sets of image and text prompts combine to form the dataset $D$, where they are fed to the model $M_i$.
• Figure 2: The complete set of images from the dataset.
• Figure 3: Zero-shot accuracy overall (%) by model, number of edges, and structure. Consistent with the feature importance analysis, accuracy decreases as the number of edges increases.
• Figure 4: Example model responses to a in-order traversal of a binary search tree. The extracted predictions are highlighted.
• Figure 5: A 3D perspective of the PCA image feature space from the first three components.