GeoEdit: Geometric Knowledge Editing for Large Language Models

TL;DR

This work tackles the challenge of updating up-to-date knowledge in large language models without eroding generalization. It introduces GeoEdit, a fine-tuning-based framework that leverages neuron-level task vectors and angular relationships to separate general-knowledge perturbations from new-knowledge updates. By applying direction-aware knowledge identification and an importance-guided task vector fusion, GeoEdit masks undesired updates and adaptively fuses useful ones, achieving superior locality while maintaining reliability and generality. Evaluations on two datasets show GeoEdit outperforming existing fine-tuning baselines and offering complementary benefits to locate-and-edit methods, with efficient training and clear ablation-backed insights into which geometric components most drive performance.

Abstract

Regular updates are essential for maintaining up-to-date knowledge in large language models (LLMs). Consequently, various model editing methods have been developed to update specific knowledge within LLMs. However, training-based approaches often struggle to effectively incorporate new knowledge while preserving unrelated general knowledge. To address this challenge, we propose a novel framework called Geometric Knowledge Editing (GeoEdit). GeoEdit utilizes the geometric relationships of parameter updates from fine-tuning to differentiate between neurons associated with new knowledge updates and those related to general knowledge perturbations. By employing a direction-aware knowledge identification method, we avoid updating neurons with directions approximately orthogonal to existing knowledge, thus preserving the model's generalization ability. For the remaining neurons, we integrate both old and new knowledge for aligned directions and apply a "forget-then-learn" editing strategy for opposite directions. Additionally, we introduce an importance-guided task vector fusion technique that filters out redundant information and provides adaptive neuron-level weighting, further enhancing model editing performance. Extensive experiments on two publicly available datasets demonstrate the superiority of GeoEdit over existing state-of-the-art methods.

Figures (4)

• Figure 1: Conceptual illustration of F-Learning ni-etal-2024-forgetting and our proposed GeoEdit.
• Figure 2: Overview of GeoEdit.Step (a): Neuron-level task vectors $\tau_{old}$ and $\tau_{new}$ are extracted for both the old and new knowledge datasets using parametric arithmetic. Step (b): An auto-encoder is trained to project a low-dimensional representation of the task vectors, eliminating the angular bias issue in high-dimensional space. Step (c): The latent task vectors, $h_{new}$ and $h_{old}$, are reduced to two dimensions using t-SNE to compute the angular relationships, which are used to classify neurons based on the angle. Finally, after applying different editing strategies, we obtain the edited vector $\tau_{edit}$, which is added to the initial model to generate the edited model $f_{\theta_{e}}$.
• Figure 3: Distribution of the angles $\phi$ between task vectors before and after dimensionality reduction.
• Figure 4: Visualization of the magnitudes of task vectors $\tau_{\text{old}}$ and $\tau_{\text{new}}$ along with the importance-guided fusion weights. All results are normalized to the range of 0 to 1.