See or Say Graphs: Agent-Driven Scalable Graph Understanding with Vision-Language Models

TL;DR

GraphVista tackles two core challenges in vision–language graph understanding: scalability to large graphs and effective coordination between text and vision modalities. It introduces a hierarchical GraphRAG base to compress and retrieve task-relevant context, a planning agent to route tasks to the appropriate modality, and Visual Graph Thoughts with process-level Direct Preference Optimization to enable grounded, stepwise visual reasoning for complex local tasks. The framework is evaluated on the Grena benchmark and GraphSQA, where GraphVista achieves up to $4.4\times$ quality improvement over baselines and scales to graphs with up to $|V|=2050$, outperforming text-only and fusion approaches on both simple and complex tasks. These results demonstrate the practical impact of combining scalable retrieval, modality-aware task routing, and grounded multimodal reasoning for robust graph understanding in real-world, large-scale settings.

Abstract

Vision-language models (VLMs) have shown promise in graph understanding, but remain limited by input-token constraints, facing scalability bottlenecks and lacking effective mechanisms to coordinate textual and visual modalities. To address these challenges, we propose GraphVista, a unified framework that enhances both scalability and modality coordination in graph understanding. For scalability, GraphVista organizes graph information hierarchically into a lightweight GraphRAG base, which retrieves only task-relevant textual descriptions and high-resolution visual subgraphs, compressing redundant context while preserving key reasoning elements. For modality coordination, GraphVista introduces a planning agent that routes tasks to the most suitable modality-using the text modality for simple property reasoning and the visual modality for local and structurally complex reasoning grounded in explicit topology. Extensive experiments demonstrate that GraphVista scales to large graphs, up to $200\times$ larger than those used in existing benchmarks, and consistently outperforms existing textual, visual, and fusion-based methods, achieving up to $4.4\times$ quality improvement over the state-of-the-art baselines by fully exploiting the complementary strengths of both modalities.

Figures (8)

• Figure 1: Comparison of textual and visual modalities across different types of graph understanding tasks
• Figure 2: Framework of GraphVista
• Figure 3: Analysis on Graphs with Different $|V|$
• Figure 4: Hyperparameter analysis on $k$ and $N_{max}$
• Figure 5: Hyperparameter analysis on storage strategy of $\mathcal{K}$