Representation Interventions Enable Lifelong Unstructured Knowledge Control

TL;DR

RILKE reframes lifelong knowledge editing as representation-space interventions rather than weight updates, enabling precise, paraphrase-robust edits with a frozen base model. It introduces a robust training objective, a query-adaptive router, and a cluster-based shared-subspace mechanism to achieve scalable, memory-efficient control over unstructured knowledge. Empirical results on LLaMA and Qwen show high edit success and strong paraphrase generalization while preserving general utility and incurring modest memory overhead. This representation-centric approach offers a practical pathway for continual knowledge updates in large language models with minimal interference across edits.

Abstract

Large language models (LLMs) often produce incorrect or outdated content. Updating their knowledge efficiently and accurately without costly retraining is a major challenge. This problem is especially hard for complex, unstructured knowledge in a lifelong setting, where many edits must coexist without interference. We introduce RILKE (Representation Intervention for Lifelong KnowledgE Control), a robust and scalable method that treats knowledge control as interventions within the model's representation space. Leveraging representation-space expressiveness, we identify two properties enabling RILKE to deliver fine-grained control over complex, unstructured knowledge while maintaining general utility with frozen base weights. During training, RILKE learns paraphrase-robust and edit-localized modules that limit each update to a low-dimensional subspace to minimize cross-edit interference. In inference, a query-adaptive router selects the appropriate module to guide the model's generation. In evaluation on knowledge editing benchmarks with LLaMA and Qwen models, RILKE is scalable to large-scale datasets, demonstrating high edit success, strong paraphrase generalization, and preserving general utility with modest memory overhead. These results show RILKE is an effective and scalable solution for lifelong knowledge control in LLMs.

Figures (6)

• Figure 1: Two key properties in LLMs’ representation space. We show that these properties enable generalizable, lifelong, and scalable knowledge control in LLMs.
• Figure 2: Overview of the RILKE framework. During training, the intervention module $\Phi$ maps $\boldsymbol{h}_{\text{ori}}$ to the target $\boldsymbol{h}_{\text{edit}}$, while $\boldsymbol{h}_{\text{ori}}$ is stored as the knowledge index. At inference, the router selects the intervention module $\Phi$ whose associated index is closest to the input query’s representation, enabling the model to generate the desired output.
• Figure 3: Edit efficacy and generalization of RILKE across layers of Llama-3.1-8B-Instruct (32 layers in total). Performance peaks when intervening in the mid-layers.
• Figure 4: Visualization of ${\mathbf{V}_{\text{edit}}}$ under different training settings. Training with similar data keeps the edited vectors close to the individual training, whereas dissimilar batching drives them away, underscoring the necessity of clustering similar knowledge for effective shared-subspace control.
• Figure 5: Knowledge Retention Rate(measured via Rouge-L) After Edits in Different Steps. We sequentially edit 100 knowledge items in batches of 10 and evaluate Knowledge Retention at each step after the final edit. All existing methods exhibit substantial forgetting of earlier edits as new knowledge accumulates.