Scalable and Accurate Graph Reasoning with LLM-based Multi-Agents

TL;DR

This work tackles the challenges of graph reasoning with large language models (LLMs), notably limited graph scale, accuracy, and lack of explicit reasoning traces when using a single LLM. It introduces GraphAgent-Reasoner (GAR), a fine-tuning-free, multi-agent framework where a Master LLM constructs a network of node-centric agents that collaboratively solve graph problems through distributed execution and formalized communication. GAR leverages a distributed algorithm library and a six-component design (State, Message, Initialization, Send, Update, Termination) to ensure transparent, stepwise reasoning and scalability up to at least 1,000 nodes, with Master summarization producing the final answer. Empirical results on GraphInstruct show near-perfect accuracy on polynomial-time tasks and robust performance on larger graphs, outperforming closed-source models and fine-tuned open-source baselines; a real-world webpage importance case demonstrates practical applicability. Overall, GAR enables scalable, explicit, and reliable graph reasoning with LLMs in real-world contexts, signaling a promising direction for distributed LLM-assisted computation on graph-structured data.

Abstract

Recent research has explored the use of Large Language Models (LLMs) for tackling complex graph reasoning tasks. However, due to the intricacies of graph structures and the inherent limitations of LLMs in handling long text, current approaches often fail to deliver satisfactory accuracy, even on small-scale graphs and simple tasks. To address these challenges, we introduce GraphAgent-Reasoner, a fine-tuning-free framework that utilizes a multi-agent collaboration strategy for explicit and precise graph reasoning. Inspired by distributed graph computation theory, our framework decomposes graph problems into smaller, node-centric tasks that are distributed among multiple agents. The agents collaborate to solve the overall problem, significantly reducing the amount of information and complexity handled by a single LLM, thus enhancing the accuracy of graph reasoning. By simply increasing the number of agents, GraphAgent-Reasoner can efficiently scale to accommodate larger graphs with over 1,000 nodes. Evaluated on the GraphInstruct dataset, our framework demonstrates near-perfect accuracy on polynomial-time graph reasoning tasks, significantly outperforming the best available models, both closed-source and fine-tuned open-source variants. Our framework also demonstrates the capability to handle real-world graph reasoning applications such as webpage importance analysis.

Figures (10)

• Figure 1: The current situation of LLMs in solving graph problems. Previous methods using a single LLM often failed due to the complex graph structures. In contrast, our approach leverages agents collaboration to effectively address graph problems.
• Figure 2: The performance of a single LLM in memorizing first-order neighboring nodes. As the number of nodes increases, all models exhibit significant memory errors.
• Figure 3: The framework of GraphAgent-Reasoner. Given a graph problem, the Master LLM will first construct agents network according to graph strcutures. It then sequentially performs Algorithm Establishing, Distributed Execution and Master Summarization, as detailed in this section.
• Figure 4: Performance of GraphAgent-Reasoner, GPT4(2 shot) and GraphWiz(Mistral 7B) on cycle detection and shortest path problems with different graph sizes.
• Figure 5: The importance analysis in webpage network. While the GraphWiz fails due to incorrect graph assessments, GAR correctly uses the PageRank algorithm to identify nodes 16, 14, and 5 as the most important.