A Dynamic Retrieval-Augmented Generation System with Selective Memory and Remembrance

TL;DR

Static non-parametric memories in retrieval-augmented generation systems limit adaptation to evolving data and user behavior.Adaptive RAG Memory (ARM) introduces a dynamic embedding layer with a Selective Remembrance and Decay policy that consolidates frequently retrieved items into a remembered set while allowing stale entries to decay, enabling continual adaptation without retraining. With approximately 22 million embedding parameters, ARM achieves competitive retrieval quality (e.g., NDCG@5 ≈ 0.940 and Recall@5 = 1.000) and superior efficiency among ultra-efficient models, while providing interpretable memory dynamics and low per-query overhead. Across natural questions, multi-hop QA, and domain-specific corpora, ARM demonstrates robust memory dynamics, favorable speed-accuracy trade-offs, and practical deployment profiles, highlighting its potential as a production-ready, neuroscience-inspired memory for RAG systems.

Abstract

We introduce \emph{Adaptive RAG Memory} (ARM), a retrieval-augmented generation (RAG) framework that replaces a static vector index with a \emph{dynamic} memory substrate governed by selective remembrance and decay. Frequently retrieved items are consolidated and protected from forgetting, while rarely used items gradually decay, inspired by cognitive consolidation and forgetting principles. On a lightweight retrieval benchmark, ARM reaches near state-of-the-art performance (e.g., NDCG@5 $\approx$ 0.940, Recall@5 $=1.000$) with only $\sim$22M parameters in the embedding layer, achieving the best efficiency among ultra-efficient models ($<$25M parameters). In addition, we compare static vs. dynamic RAG combinations across Llama 3.1 and GPT-4o. Llama 3.1 with static RAG achieves the highest key-term coverage (67.2\%) at moderate latency, while GPT-4o with a dynamic selective retrieval policy attains the fastest responses (8.2s on average) with competitive coverage (58.7\%). We further present an engineering optimization of the DynamicRAG implementation, making embedding weights configurable, adjustable at runtime, and robust to invalid settings. ARM yields competitive accuracy, self-regularizing memory growth, and interpretable retention dynamics without retraining the generator\color{black} and provides practical trade-off between quality, latency and memory efficiency for production and research RAG system.

Figures (2)

• Figure 1: Adaptive RAG Memory pipeline with selective remembrance and decay.
• Figure 2: Conceptual dynamics of embedding strength under decay (unremembered) vs. remembrance (consolidated) as queries progress.