Lifelong Knowledge Editing for LLMs with Retrieval-Augmented Continuous Prompt Learning

TL;DR

Lifelong model editing for LLMs is challenged by catastrophic forgetting and inefficiency. The authors propose RECIPE, which uses Knowledge Retrieval Repository encoded as continuous prompts and a Knowledge Sentinel to dynamically gate retrieval, enabling on-the-fly editing without changing model parameters. Training jointly optimizes editing and prompt-learning losses, with the LLM kept frozen; results show strong lifelong editing performance and preserved general abilities across multiple backbones. RECIPE achieves faster editing and inference by using short prompts and effective retrieval, addressing both efficacy and efficiency. The work highlights the potential of retrieval-augmented continuous prompts for scalable, maintenance-friendly LLM knowledge updates.

Abstract

Model editing aims to correct outdated or erroneous knowledge in large language models (LLMs) without the need for costly retraining. Lifelong model editing is the most challenging task that caters to the continuous editing requirements of LLMs. Prior works primarily focus on single or batch editing; nevertheless, these methods fall short in lifelong editing scenarios due to catastrophic knowledge forgetting and the degradation of model performance. Although retrieval-based methods alleviate these issues, they are impeded by slow and cumbersome processes of integrating the retrieved knowledge into the model. In this work, we introduce RECIPE, a RetriEval-augmented ContInuous Prompt lEarning method, to boost editing efficacy and inference efficiency in lifelong learning. RECIPE first converts knowledge statements into short and informative continuous prompts, prefixed to the LLM's input query embedding, to efficiently refine the response grounded on the knowledge. It further integrates the Knowledge Sentinel (KS) that acts as an intermediary to calculate a dynamic threshold, determining whether the retrieval repository contains relevant knowledge. Our retriever and prompt encoder are jointly trained to achieve editing properties, i.e., reliability, generality, and locality. In our experiments, RECIPE is assessed extensively across multiple LLMs and editing datasets, where it achieves superior editing performance. RECIPE also demonstrates its capability to maintain the overall performance of LLMs alongside showcasing fast editing and inference speed.

Figures (4)

• Figure 1: Comparison among three types of methods in lifelong editing scenarios. Modifying parameters and adding extra parameters result in the degradation of LLM performance as editing progresses. In contrast, retrieval-based editors store knowledge in a repository and apply knowledge editing on the fly, which maintains the LLM unchanged and relieves it from accumulating parameter offsets or adding extra parameters. (Best viewed in clolor)
• Figure 2: Illustration of the RECIPE framework. Process 1 constructs and updates the knowledge retrieval repository $\mathcal{K}_t$. During the inference stage, Process 2 retrieves query-related prompts from $\mathcal{K}_t$. Process 3 utilizes the retrieved continuous prompts to correct the LLM's response. For lifelong editing, the repository can be continuously updated (e.g., from $\mathcal{K}_{t-1}$ to $\mathcal{K}_{t}$) with each new insertion of knowledge and prompts.
• Figure 3: Impact of the number of CPTs on editing performance of RECIPE.
• Figure 4: Visualization of word embeddings with varying numbers of CPTs.