Scaling Graph Chain-of-Thought Reasoning: A Multi-Agent Framework with Efficient LLM Serving

TL;DR

GLM tackles the scalability challenges of Graph-CoT by introducing a multi-agent reasoning framework that decouples retrieval from reasoning and couples it with a Graph-CoT–aware inference stack. The approach replaces long multi-hop CoT chains with task-specific code generation and a vertex-centric KV-cache strategy, achieving higher accuracy while dramatically reducing token usage, latency, and increasing throughput. Extensive evaluation on GRBench demonstrates substantial gains over baselines, including up to 60% accuracy improvements and up to 15.1× throughput increases, validating the practicality of scalable, structured LLM reasoning over graphs. This work highlights the importance of system-level co-design in enabling efficient, real-world graph-aware reasoning with LLMs.

Abstract

Graph Chain-of-Thought (Graph-CoT) enables large language models (LLMs) to perform step-by-step reasoning over graph-structured knowledge, but existing pipelines suffer from low accuracy, excessive token usage, high latency, and low throughput due to single-agent monolithic prompts, repeated context re-encoding, and inefficient serving execution. We present GLM, the first multi-agent Graph-CoT system co-designed with an optimized LLM serving architecture. GLM decomposes reasoning into specialized agents for classification, reasoning, action generation, and graph retrieval, enabling branching and selective context sharing to reduce prompt length and reasoning iterations while preserving reasoning quality, thereby improving accuracy and reducing overall token consumption. To scale inference, we introduce a Graph-CoT-aware LLM inference mechanism with graph-specific KV-cache management, priority-based eviction, and pipelined execution to improve serving efficiency. Experiments demonstrate that GLM improves answer accuracy by up to 38%, reduces token cost by up to 95.7%, lowers inference latency by 90.3%, and achieves up to 15.1x higher throughput compared to state-of-the-art Graph-CoT baselines, enabling efficient adoption for complex real-world reasoning at scale.

Figures (15)

• Figure 1: Graph-CoT Framework.
• Figure 2: Sensitivity of accuracy on reason steps.
• Figure 3: Multi-agent reasoning framework workflow.
• Figure 4: Graph-CoT and GLM's agents comparison.
• Figure 5: Multi-agent reasoning example.

Theorems & Definitions (1)

• Definition 1: LLM-based Agent