Reasoning with Graphs: Structuring Implicit Knowledge to Enhance LLMs Reasoning

TL;DR

This work addresses the difficulty LLMs face in complex reasoning tasks by converting implicit context into explicit graphs. It introduces Reasoning with Graphs (RwG), a two-stage approach that iteratively constructs a graph from the problem context and then reasons over this graph to answer questions, without relying on external graphs. Across logical reasoning and multi-hop QA tasks, RwG improves accuracy for multiple LLMs, with additional gains when explicit relations are provided and when combined with other prompting strategies. The findings demonstrate that structured, graph-based representations of contextual knowledge can enhance LLM reasoning in a general, prompt-driven manner, supporting more reliable multi-step inference in diverse domains.

Abstract

Large language models (LLMs) have demonstrated remarkable success across a wide range of tasks; however, they still encounter challenges in reasoning tasks that require understanding and inferring relationships between distinct pieces of information within text sequences. This challenge is particularly pronounced in tasks involving multi-step processes, such as logical reasoning and multi-hop question answering, where understanding implicit relationships between entities and leveraging multi-hop connections in the given context are crucial. Graphs, as fundamental data structures, explicitly represent pairwise relationships between entities, thereby offering the potential to enhance LLMs' reasoning capabilities. External graphs have proven effective in supporting LLMs across multiple tasks. However, in many reasoning tasks, no pre-existing graph structure is provided. Can we structure implicit knowledge derived from context into graphs to assist LLMs in reasoning? In this paper, we propose Reasoning with Graphs (RwG) by first constructing explicit graphs from the context and then leveraging these graphs to enhance LLM reasoning performance on reasoning tasks. Extensive experiments demonstrate the effectiveness of the proposed method in improving both logical reasoning and multi-hop question answering tasks.

Figures (5)

• Figure 1: Comparison of Reasoning with Graph (RwG) to other prompting methods. Gray nodes represent thoughts generated by LLMs, while green nodes represent entities extracted from the input.
• Figure 2: The procedure of RwG for the AIW+ example. Blue nodes represent entities explicitly mentioned in the context and included in the initial graph, while red nodes denote inferred entities added during the graph generation and verification processes. The node names are based on their relationship to Alice.
• Figure 3: Comparison of performances under different verification steps.
• Figure 4: Performance on different hop questions in the Clutrr dataset.
• Figure 5: The illustration of Case 2.