Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Energy-Regularized Sequential Model Editing on Hyperspheres

Qingyuan Liu, Jia-Chen Gu, Yunzhi Yao, Hong Wang, Nanyun Peng

TL;DR

This paper addresses the instability and knowledge degradation observed during sequential updates to large language models. It introduces hyperspherical uniformity and Hyperspherical Energy (HE) as diagnostic tools and demonstrates a strong link between HE dynamics and editing stability, supported by a theoretical bound that connects HE changes to knowledge degradation. The authors propose SPHERE, an HE-driven regularization that projects edits into a sparse subspace orthogonal to the principal directions of pretrained weights, thereby preserving prior knowledge while enabling reliable sequential edits. Empirical results on LLaMA3-8B and Qwen2.5-7B show SPHERE outperforms state-of-the-art baselines in editing capability and general ability retention, and it also provides a plug-and-play boost to existing methods, offering a principled path toward scalable, reliable knowledge editing.

Abstract

Large language models (LLMs) require constant updates to remain aligned with evolving real-world knowledge. Model editing offers a lightweight alternative to retraining, but sequential editing often destabilizes representations and induces catastrophic forgetting. In this work, we seek to better understand and mitigate performance degradation caused by sequential editing. We hypothesize that hyperspherical uniformity, a property that maintains uniform distribution of neuron weights on a hypersphere, helps the model remain stable, retain prior knowledge, while still accommodate new updates. We use Hyperspherical Energy (HE) to quantify neuron uniformity during editing, and examine its correlation with editing performance. Empirical studies across widely used editing methods reveals a strong correlation between HE dynamics and editing performance, with editing failures consistently coinciding with high HE fluctuations. We further theoretically prove that HE dynamics impose a lower bound on the degradation of pretrained knowledge, highlighting why HE stability is crucial for knowledge retention. Motivated by these insights, we propose SPHERE (Sparse Projection for Hyperspherical Energy-Regularized Editing), an HE-driven regularization strategy that stabilizes neuron weight distributions, ultimately preserving prior knowledge while enabling reliable sequential updates. Specifically, SPHERE identifies a sparse space complementary to the principal hyperspherical directions of the pretrained weight matrices and projects new knowledge onto it, attenuating perturbations on the principal directions. Extensive experiments on LLaMA3 (8B) and Qwen2.5 (7B) show that SPHERE outperforms the best baseline in editing capability by an average of 16.41%, while most faithfully preserving general model performance, thereby offering a principled path toward reliable large-scale knowledge editing.

Energy-Regularized Sequential Model Editing on Hyperspheres

TL;DR

This paper addresses the instability and knowledge degradation observed during sequential updates to large language models. It introduces hyperspherical uniformity and Hyperspherical Energy (HE) as diagnostic tools and demonstrates a strong link between HE dynamics and editing stability, supported by a theoretical bound that connects HE changes to knowledge degradation. The authors propose SPHERE, an HE-driven regularization that projects edits into a sparse subspace orthogonal to the principal directions of pretrained weights, thereby preserving prior knowledge while enabling reliable sequential edits. Empirical results on LLaMA3-8B and Qwen2.5-7B show SPHERE outperforms state-of-the-art baselines in editing capability and general ability retention, and it also provides a plug-and-play boost to existing methods, offering a principled path toward scalable, reliable knowledge editing.

Abstract

Large language models (LLMs) require constant updates to remain aligned with evolving real-world knowledge. Model editing offers a lightweight alternative to retraining, but sequential editing often destabilizes representations and induces catastrophic forgetting. In this work, we seek to better understand and mitigate performance degradation caused by sequential editing. We hypothesize that hyperspherical uniformity, a property that maintains uniform distribution of neuron weights on a hypersphere, helps the model remain stable, retain prior knowledge, while still accommodate new updates. We use Hyperspherical Energy (HE) to quantify neuron uniformity during editing, and examine its correlation with editing performance. Empirical studies across widely used editing methods reveals a strong correlation between HE dynamics and editing performance, with editing failures consistently coinciding with high HE fluctuations. We further theoretically prove that HE dynamics impose a lower bound on the degradation of pretrained knowledge, highlighting why HE stability is crucial for knowledge retention. Motivated by these insights, we propose SPHERE (Sparse Projection for Hyperspherical Energy-Regularized Editing), an HE-driven regularization strategy that stabilizes neuron weight distributions, ultimately preserving prior knowledge while enabling reliable sequential updates. Specifically, SPHERE identifies a sparse space complementary to the principal hyperspherical directions of the pretrained weight matrices and projects new knowledge onto it, attenuating perturbations on the principal directions. Extensive experiments on LLaMA3 (8B) and Qwen2.5 (7B) show that SPHERE outperforms the best baseline in editing capability by an average of 16.41%, while most faithfully preserving general model performance, thereby offering a principled path toward reliable large-scale knowledge editing.

Paper Structure

This paper contains 55 sections, 5 theorems, 70 equations, 12 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Model Editing
  4. Hyperspherical Energy
  5. Correlation between Hyperspherical Uniformity and Editing
  6. Empirical Analysis of the HE–Editing Stability Correlation
  7. Observation 1: Collapse in sequential editing is closely tied to sharp fluctuations in HE.
  8. Observation 2: Advanced editing methods suppress HE fluctuations effectively.
  9. Theoretical Analysis of HE's Impact on Editing Stability
  10. SPHERE
  11. Principal Space Estimation
  12. Sparse Space Definition
  13. Sparse Space Projection
  14. Experiments
  15. Experimental Setup
  16. ...and 40 more sections

Key Result

Theorem 1

Under the assumptions of orthonormal inputs and small perturbations, the output perturbation $\Delta \bm{V}$ is lower-bounded by squared change in HE: where $K$ is a constant dependent on the original weight matrix geometry.

Figures (12)

  • Figure 1: (a) A weight matrix is viewed as a set of neurons (red dots) on a hypersphere. (b) Current SOTA methods DBLP:conf/iclr/MaWXLG25DBLP:conf/iclr/FangJWMSW0C25 introduce perturbations (blue triangles) that interfere with the principle hyperspherical directions of pre-edit weights. (c) SPHERE projects new knowledge onto a sparse space complementary to the principal hyperspherical directions.
  • Figure 2: Trends of HE and editing performance throughout sequential editing. The Spearman correlation scores between HE and each editing metric displayed at the end of each curve.
  • Figure 3: Correlation between changes in HE and editing performance across consecutive edited weights. Each point corresponds to a $\Delta \mathrm{HE}$–$\Delta \mathrm{Acc.}$ pair for one method over five thousand sequential edits. Confidence ellipses and regression lines illustrate overall trends.
  • Figure 4: Cosine similarity between neurons in the updated weight matrix after 15,000 edits. Darker colors indicate lower similarity, reflecting better hyperspherical and orthogonal uniformity. SPHERE effectively preserves the weight structure, demonstrating the most stable hyperspherical uniformity.
  • Figure 5: The t-SNE distribution of weight neurons of pre-edited and post-edited LLM after 15,000 edits using dimensionality reduction. The top and right curve graphs display the marginal distributions for two reduced dimensions, where SPHERE consistently exhibits minimal shift.
  • ...and 7 more figures

Theorems & Definitions (6)

  • Theorem 1: Lower Bound on Output Perturbation
  • Theorem 2: Upper Bound on HE Change
  • proof
  • Lemma 1
  • Theorem 3
  • Theorem 4