Bridging Visualization and Optimization: Multimodal Large Language Models on Graph-Structured Combinatorial Optimization

TL;DR

This work reimagines graph-structured combinatorial optimization by converting graphs into images and leveraging multimodal large language models (MLLMs) to perform complex reasoning without fine-tuning. Through simple prompts and a lightweight local search, the approach yields competitive or superior results for influence maximization and network dismantling, across both small and large networks, while maintaining accuracy on fundamental graph tasks. The key contributions include an image-based graph representation that preserves high-order structure, labeling and layout strategies to support visual reasoning, and demonstrated scalability to large, real-world networks with partial-label visualizations. The findings suggest a practical, scalable paradigm for graph reasoning with MLLMs, motivating future integration of interactive visualization tools to further unlock performance on massive networks.

Abstract

Graph-structured combinatorial challenges are inherently difficult due to their nonlinear and intricate nature, often rendering traditional computational methods ineffective or expensive. However, these challenges can be more naturally tackled by humans through visual representations that harness our innate ability for spatial reasoning. In this study, we propose transforming graphs into images to preserve their higher-order structural features accurately, revolutionizing the representation used in solving graph-structured combinatorial tasks. This approach allows machines to emulate human-like processing in addressing complex combinatorial challenges. By combining the innovative paradigm powered by multimodal large language models (MLLMs) with simple search techniques, we aim to develop a novel and effective framework for tackling such problems. Our investigation into MLLMs spanned a variety of graph-based tasks, from combinatorial problems like influence maximization to sequential decision-making in network dismantling, as well as addressing six fundamental graph-related issues. Our findings demonstrate that MLLMs exhibit exceptional spatial intelligence and a distinctive capability for handling these problems, significantly advancing the potential for machines to comprehend and analyze graph-structured data with a depth and intuition akin to human cognition. These results also imply that integrating MLLMs with simple optimization strategies could form a novel and efficient approach for navigating graph-structured combinatorial challenges without complex derivations, computationally demanding training and fine-tuning.

Figures (17)

• Figure 1: The representation of different eras of graph structure. (a) Adjacency matrix; (b) Embedding; (c) Text; (d) Image.
• Figure 2: Visualization of different networks with their original community structure and corresponding merged structure, each displayed using the Fruchterman-Reingold Layout. The number of original and merged communities for each network is as follows: Facebook (reduced from 13 to 10 communities), Router (reduced from 63 to 10 communities), and Sex (reduced from 170 to 9 communities).
• Figure 3: The illustrations of three agents for IM on small-scale networks. The full-label network (left) will be inputted into MLLM along with the prompts for agents (right).
• Figure 4: The comparative IM performance on small-scale networks with the IC and LT models.
• Figure 5: The IM result of MLLM agents with and without local search on small-scale networks. LS refers to local search.