FragRel: Exploiting Fragment-level Relations in the External Memory of Large Language Models

TL;DR

This work tackles the limitation of fixed context windows in LLMs by introducing fragment-level relations within an external memory framework. It proposes a relation-aware fragment assessment that combines independent fragment relevance with environment-informed scores derived from semantic, contextual, and code-structure relations, enabling a fragment-connected Hierarchical Memory (HieraMem-LLM). Through extensive experiments on long-story understanding, repository-level code generation, and memory-enhanced chatting, the approach shows consistent improvements over baselines, particularly with longer contexts and when using structure-based relations. The findings underscore the practical impact of encoding cross-fragment relations to better leverage external memory for long-context tasks, while acknowledging limitations around manually defined relations and parameter choices that invite future automation and broader validation.

Abstract

To process contexts with unlimited length using Large Language Models (LLMs), recent studies explore hierarchically managing the long text. Only several text fragments are taken from the external memory and passed into the temporary working memory, i.e., LLM's context window. However, existing approaches isolatedly handle the text fragments without considering their structural connections, thereby suffering limited capability on texts with intensive inter-relations, e.g., coherent stories and code repositories. This work attempts to resolve this by exploiting the fragment-level relations in external memory. First, we formulate the fragment-level relations and present several instantiations for different text types. Next, we introduce a relation-aware fragment assessment criteria upon previous independent fragment assessment. Finally, we present the fragment-connected Hierarchical Memory based LLM. We validate the benefits of involving these relations on long story understanding, repository-level code generation, and long-term chatting.

Figures (4)

• Figure 1: The instantiated fragment-level relations in various text types.
• Figure 2: Overall framework of Fragment-connected Hierarchical Memory based LLM. First, the long context (complete book, code repository, or agent memory) is split into a lot of fragments $c_*$. Then we calculate the independent similarity score $score^{ind}_*$ between every fragment and the user question. Next, for every fragment, its relation-aware score $score^{rel}_*$ is calculated as the combination of the independent score and the normalized relation-weighted summation of other fragments' independent score. Finally, the fragments with top $K$ relation-aware scores are retrieved for LLM inference.
• Figure 3: Performance comparison on NarrativeQA kovcisky2018narrativeqa. The horizontal lines and columnas represent Temporary Memory based LLMs and Hierarchical Memory based LLMs respectively. The Hierarchical Memory based LLMs contain 3 categories: no relation incorporation, semantic relation incorporation and context structure relation incorporation, denoted as "No-FragRel-*", "A-FragRel-*", and "B-FragRel-*" respectively.
• Figure 4: Performance improvement using different edge weight $w^{rel}$ and relation weight $\alpha$.