Improving Multi-step RAG with Hypergraph-based Memory for Long-Context Complex Relational Modeling

TL;DR

HGMem tackles the difficulty of global sense-making in long-context, multi-step RAG by replacing static memory with a hypergraph-based memory that evolves to encode high-order relations. The approach represents memory as a hypergraph whose hyperedges act as memory points, and memory evolves through Update, Insertion, and Merging while performing adaptive local/global evidence retrieval. The paper provides formal problem formulation, a detailed methodology for memory storage and retrieval, and extensive experiments on generative sense-making QA and long narrative understanding demonstrating consistent improvements over strong baselines, including competitive performance against GPT-4o with a smaller model. This work advances long-context reasoning by enabling structured, dynamic memory that guides iterative retrieval and reasoning, with practical implications for scalable, interpretable RAG systems.

Abstract

Multi-step retrieval-augmented generation (RAG) has become a widely adopted strategy for enhancing large language models (LLMs) on tasks that demand global comprehension and intensive reasoning. Many RAG systems incorporate a working memory module to consolidate retrieved information. However, existing memory designs function primarily as passive storage that accumulates isolated facts for the purpose of condensing the lengthy inputs and generating new sub-queries through deduction. This static nature overlooks the crucial high-order correlations among primitive facts, the compositions of which can often provide stronger guidance for subsequent steps. Therefore, their representational strength and impact on multi-step reasoning and knowledge evolution are limited, resulting in fragmented reasoning and weak global sense-making capacity in extended contexts. We introduce HGMem, a hypergraph-based memory mechanism that extends the concept of memory beyond simple storage into a dynamic, expressive structure for complex reasoning and global understanding. In our approach, memory is represented as a hypergraph whose hyperedges correspond to distinct memory units, enabling the progressive formation of higher-order interactions within memory. This mechanism connects facts and thoughts around the focal problem, evolving into an integrated and situated knowledge structure that provides strong propositions for deeper reasoning in subsequent steps. We evaluate HGMem on several challenging datasets designed for global sense-making. Extensive experiments and in-depth analyses show that our method consistently improves multi-step RAG and substantially outperforms strong baseline systems across diverse tasks.

Figures (10)

• Figure 1: (i) The RAG system at its $t$-th interaction step. ①: The LLM adaptively generates a set of subqueries $\mathcal{Q}^{(t)}$ for either local investigation or global exploration (see Section \ref{['subsec:retrieval_strategy']}). ②: $\mathcal{Q}^{(t)}$ are used to retrieve information from $\mathcal{D}$ and $\mathcal{G}$. ③: $\mathcal{V}_{\mathcal{Q}^{(t)}}$, $\mathcal{E}(\mathcal{V}_{\mathcal{Q}^{(t)}})$ and $\mathcal{D}(\mathcal{V}_{\mathcal{Q}^{(t)}})$ are obtained through graph-based indexing and vector-based matching. ④: The LLM evolves current memory $\mathcal{M}^{(t)}$ into $\mathcal{M}^{(t+1)}$ using Equation \ref{['eq:memory_evolving']}. (ii) The structure of our proposed hypergraph-based memory that evolves through update, insertion and merging operations.
• Figure 2: An illustration of memory evolving dynamics. Each point is equivalent to a hyperedge in the hypergraph. $\mathcal{M}^{(t)}$ evolves into $\mathcal{M}^{(t+1)}$ through update, insertion and merging operations.
• Figure 3: Prediction accuracies at different steps using Qwen2.5-32B-Instruct on long narrative understanding datasets.
• Figure 4: The prompt for updating and inserting memory points during memory evolving in HGMem.
• Figure 5: The prompt for merging memory points during memory evolving in HGMem.