Graph-Grounded LLMs: Leveraging Graphical Function Calling to Minimize LLM Hallucinations

TL;DR

This work tackles LLM hallucinations in graph reasoning by grounding LLMs with a dedicated graph library accessed through function calls, enabling iterative graph construction and operation execution. The approach, Graph-Grounded LLMs, leverages a 29-function library built on networkx and multi-round closed-loop reasoning to reduce errors and improve scalability. Evaluated on the NLGraph benchmark (6022 problems across eight graph tasks), the method achieves near-100% accuracy on many tasks and demonstrates strong performance in a disaster-response scenario as a natural-language decision-support system. The results indicate significant advances for reliable graph-aware AI suitable for safety-critical applications, with future work on retrieval augmentation and automated library evolution to enhance scalability and robustness.

Abstract

The adoption of Large Language Models (LLMs) is rapidly expanding across various tasks that involve inherent graphical structures. Graphs are integral to a wide range of applications, including motion planning for autonomous vehicles, social networks, scene understanding, and knowledge graphs. Many problems, even those not initially perceived as graph-based, can be effectively addressed through graph theory. However, when applied to these tasks, LLMs often encounter challenges, such as hallucinations and mathematical inaccuracies. To overcome these limitations, we propose Graph-Grounded LLMs, a system that improves LLM performance on graph-related tasks by integrating a graph library through function calls. By grounding LLMs in this manner, we demonstrate significant reductions in hallucinations and improved mathematical accuracy in solving graph-based problems, as evidenced by the performance on the NLGraph benchmark. Finally, we showcase a disaster rescue application where the Graph-Grounded LLM acts as a decision-support system.

Figures (4)

• Figure 1: Schematic of the Graph-Grounded LLM. A graph library containing various graph functions is created, with each function described in JSON format and provided to the LLM as tools alongside the context and graph problem description. The LLM employs chain-of-thought reasoning to devise a solution strategy and determine the necessary function calls. These function calls are then executed by the LLM function manager, which returns the computed outputs to the LLM for further reasoning and problem-solving.
• Figure 2: JSON description of the find_shortest_distance function for LLM.
• Figure 3: Overview of NLGraph Benchmark Tasks: This illustration presents eight graph-related tasks of varying complexity, each accompanied by sample natural language prompts used with LLMs. The image is recreated from wang2024can.
• Figure 4: Illustration of the disaster response application with some fire locations, building collapse, victims and rescue robots. The Graph-Grounded LLM acts as a natural language decision support system for the disaster response mission.

Theorems & Definitions (1)

• Remark 1