FLEKE: Federated Locate-then-Edit Knowledge Editing

TL;DR

This paper defines Federated Locate-then-Edit Knowledge Editing (FLEKE), a framework for multi-client, privacy-preserving knowledge editing of large language models, addressing inefficiencies from redundant MKV computations in decentralized environments. It introduces FedEdit, a two-stage process where clients locally perform LEKE to produce mediator knowledge vectors (MKVs), upload them to a server, and later re-edit by retrieving server-stored MKVs via cosine similarity to minimize recomputation. Experimental results on reorganized zsRE and COUNTERFACT datasets using GPT-J (6B) and GPT-NeoX (20B) show that FedEdit achieves at least 96% of centralized LEKE performance and significantly outperforms FedAvg baselines, with MEMIT yielding more consistent results than PMET under FedEdit. The work demonstrates robust improvements in reliability and generalization in federated settings and provides a practical path for scalable, privacy-conscious knowledge editing across organizations such as hospitals and financial institutions.

Abstract

Locate-then-Edit Knowledge Editing (LEKE) is a key technique for updating large language models (LLMs) without full retraining. However, existing methods assume a single-user setting and become inefficient in real-world multi-client scenarios, where decentralized organizations (e.g., hospitals, financial institutions) independently update overlapping knowledge, leading to redundant mediator knowledge vector (MKV) computations and privacy concerns. To address these challenges, we introduce Federated Locate-then-Edit Knowledge Editing (FLEKE), a novel task that enables multiple clients to collaboratively perform LEKE while preserving privacy and reducing computational overhead. To achieve this, we propose FedEdit, a two-stage framework that optimizes MKV selection and reuse. In the first stage, clients locally apply LEKE and upload the computed MKVs. In the second stage, rather than relying solely on server-based MKV sharing, FLEKE allows clients retrieve relevant MKVs based on cosine similarity, enabling knowledge re-edit and minimizing redundant computations. Experimental results on two benchmark datasets demonstrate that FedEdit retains over 96% of the performance of non-federated LEKE while significantly outperforming a FedAvg-based baseline by approximately twofold. Besides, we find that MEMIT performs more consistently than PMET in the FLEKE task with our FedEdit framework. Our code is available at https://github.com/zongkaiz/FLEKE.

Figures (6)

• Figure 1: Comparison between (a) non-federated LEKE and (b) the proposed FLEKE task, where the former requires the computation of the mediator knowledge vector multiple times for the same knowledge through gradient descent, while the latter computes it only once.
• Figure 2: Cosine similarity between core text in zsRE dataset and the corresponding $z_i$ vectors.
• Figure 3: The overview of the classic LEKE method named MEMIT r2.
• Figure 4: The workflow of the proposed FedEdit.
• Figure 5: The editing performance of FedEdit and baselines with the number of clients. The suffix "w/o" indicates the Ablation experimental group.