MindBridge: Scalable and Cross-Model Knowledge Editing via Memory-Augmented Modality

TL;DR

MindBridge tackles the problem of cross-model knowledge editing, where edited knowledge should persist across updates to different LLM backbones. It introduces memory modality as an independent knowledge carrier and a two-stage process: memory modality pre-training (inject, associate, exist) and memory modality bridging (a lightweight projector) to enable cross-model grounding of edited facts. Empirical results on ZsRE and Counterfact across GPT2-XL, GPT-J, and LLaMA3 demonstrate superior editing performance, scalability to tens of thousands of edits, and the ability to bridge memory across multiple LLMs, outperforming various baselines. The work offers a practical pathway to maintain up-to-date, personalized, or domain-specific knowledge without repeatedly re-editing every model update, with code released for community use.

Abstract

Knowledge editing is a technique for efficiently and accurately updating the knowledge of large language models (LLMs) to alleviate obsolescence and correct errors. However, most existing methods overfit to specific models, causing edited knowledge to be discarded during each LLM update and requiring frequent re-editing, which is particularly burdensome in today's rapidly evolving open-source community. To address this issue, we propose the problem of cross-model knowledge editing and introduce MindBridge, a scalable solution inspired by the low coupling between modality processing and LLMs in multi-modal models. MindBridge introduces the novel concept of memory modality, which encodes edited knowledge as an independent modality. It first performs LLM-agnostic pre-training of the memory modality and then integrates it with various LLMs. Extensive experiments on multiple LLMs and popular knowledge editing datasets demonstrate that MindBridge achieves superior performance even in editing tens of thousands of knowledge entries and can flexibly adapt to different LLMs. Our code is available at https://github.com/CrashBugger/MindBridge.

Figures (8)

• Figure 1: (a) The cross-model knowledge editing problem. Current knowledge editing methods discard previously edited knowledge after every LLM update (e.g., when the base model is updated alongside LLaMA), requiring frequent re-editing, which is labor-intensive. This motivates us to explore whether edited knowledge can transcend individual models, i.e., achieve cross-model knowledge editing. (b) Analogizing memory modality to visual modality. Different visual modality encoders exhibit low coupling with various LLM backbones, and after efficient modality alignment, they enable LLMs to see. Inspired by this, we propose memory modality, which decouples knowledge from a single model, allowing knowledge editing through modality bridging with different LLMs.
• Figure 2: Overview of MindBridge. Given a massive collection of fact knowledge subject-relation-object triplets $(s, r, o^*)$ intended for editing, we first perform stage I: Memory Modality Pre-training. This phase utilizes three training objectives -- memory injection, memory association, and memory existence -- to develop a memory modality encoder capable of retaining relevant memories, making associations and determining whether specific memories exist. In stage II: Memory Modality Bridging, we then train a projector to bridge the memory modality encoder with LLMs, allowing these models to obtain and effectively leverage the relevant memories.
• Figure 3: F1-score of LLaMA3 (8B) on the GLUE benchmark after editing 10,000 facts using MindBridge, AlphaEdit, and MEMIT. The evaluation includes six tasks: SST, MRPC, CoLA, RTE, MMLU, and NLI.
• Figure 4: Visualization of the dimensionality-reduced distributions of representations extracted by the memory modality encoder for $\mathcal{I}$ and $\mathcal{O}$.
• Figure 5: Scaling MindBridge to 60,000 edits on ZsRE.