Memory Bank Compression for Continual Adaptation of Large Language Models

TL;DR

This work tackles scalable continual adaptation of large language models by introducing Memory Bank Compression (MBC). MBC compresses the external memory via a VQ-VAE-inspired codebook, uses an online EMA-based codebook resetting to prevent collapse, and applies Key-Value Low-Rank Adaptation to modulate attention without full fine-tuning. The architecture combines an amortization network, a compressed memory bank, and an aggregation network to generate query-specific modulations that condition the base LLM. Experiments on StreamingQA, SQuAD, and ArchivalQA show that MBC reduces memory usage by up to ~99% compared with strong baselines while maintaining or improving QA accuracy and retention during online adaptation. The results demonstrate that MBC provides scalable, efficient continual adaptation for diverse LLMs with practical implications for streaming data settings.

Abstract

Large Language Models (LLMs) have become a mainstay for many everyday applications. However, as data evolve their knowledge quickly becomes outdated. Continual learning aims to update LLMs with new information without erasing previously acquired knowledge. Although methods such as full fine-tuning can incorporate new data, they are computationally expensive and prone to catastrophic forgetting, where prior knowledge is overwritten. Memory-augmented approaches address this by equipping LLMs with a memory bank, that is an external memory module which stores information for future use. However, these methods face a critical limitation, in particular, the memory bank constantly grows in the real-world scenario when large-scale data streams arrive. In this paper, we propose MBC, a model that compresses the memory bank through a codebook optimization strategy during online adaptation learning. To ensure stable learning, we also introduce an online resetting mechanism that prevents codebook collapse. In addition, we employ Key-Value Low-Rank Adaptation in the attention layers of the LLM, enabling efficient utilization of the compressed memory representations. Experiments with benchmark question-answering datasets demonstrate that MBC reduces the memory bank size to 0.3% when compared against the most competitive baseline, while maintaining high retention accuracy during online adaptation learning. Our code is publicly available at https://github.com/Thomkat/MBC.

Figures (2)

• Figure 1: Memory bank footprint (logMB) of MAC and MBC across StreamingQA, SQuAD, and ArchivalQA.
• Figure 2: Perplexity over train epochs on StreamingQA with and without codebook resetting in MBC, across all base LLMs.