Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Orthogonal Residual Spread for Precise Massive Editing in Large Language Models

Xiaojie Gu, Guangxu Chen, Yuheng Yang, Jingxin Han, Andi Zhang

TL;DR

This work addresses the challenge of updating knowledge in large language models at scale without retraining. It introduces HORSE, a method that performs hierarchical orthogonal residual spread of the residual matrix across transformer layers, guided by a token-level update and a hypernetwork that predicts residual directions. The approach yields state-of-the-art results on zsRE and CounterFact across multiple backbones, with an average gain of $+6.26\%$ and notably $+10.12\%$ in Specificity, while also achieving faster editing speeds. By explicitly modeling residual propagation and layer-wise contributions, HORSE reduces interference with existing knowledge and demonstrates robust effectiveness for safe, scalable model editing in real-world scenarios.

Abstract

Large language models (LLMs) exhibit exceptional performance across various domains, yet they face critical safety concerns. Model editing has emerged as an effective approach to mitigate these issues. Existing model editing methods often focus on optimizing an information matrix that blends new and old knowledge. While effective, these approaches can be computationally expensive and may cause conflicts. In contrast, we shift our attention to Hierarchical Orthogonal Residual SprEad of the information matrix, which reduces noisy gradients and enables more stable edits from a different perspective. We demonstrate the effectiveness of our method HORSE through a clear theoretical comparison with several popular methods and extensive experiments conducted on two datasets across multiple LLMs. The results show that HORSE maintains precise massive editing across diverse scenarios. The code is available at https://github.com/XiaojieGu/HORSE

Hierarchical Orthogonal Residual Spread for Precise Massive Editing in Large Language Models

TL;DR

This work addresses the challenge of updating knowledge in large language models at scale without retraining. It introduces HORSE, a method that performs hierarchical orthogonal residual spread of the residual matrix across transformer layers, guided by a token-level update and a hypernetwork that predicts residual directions. The approach yields state-of-the-art results on zsRE and CounterFact across multiple backbones, with an average gain of and notably in Specificity, while also achieving faster editing speeds. By explicitly modeling residual propagation and layer-wise contributions, HORSE reduces interference with existing knowledge and demonstrates robust effectiveness for safe, scalable model editing in real-world scenarios.

Abstract

Large language models (LLMs) exhibit exceptional performance across various domains, yet they face critical safety concerns. Model editing has emerged as an effective approach to mitigate these issues. Existing model editing methods often focus on optimizing an information matrix that blends new and old knowledge. While effective, these approaches can be computationally expensive and may cause conflicts. In contrast, we shift our attention to Hierarchical Orthogonal Residual SprEad of the information matrix, which reduces noisy gradients and enables more stable edits from a different perspective. We demonstrate the effectiveness of our method HORSE through a clear theoretical comparison with several popular methods and extensive experiments conducted on two datasets across multiple LLMs. The results show that HORSE maintains precise massive editing across diverse scenarios. The code is available at https://github.com/XiaojieGu/HORSE
Paper Structure (11 sections, 7 equations, 2 figures, 3 tables)

This paper contains 11 sections, 7 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Preliminary
  4. Residual Matrix Optimization
  5. HyperNetwork Training
  6. Experiments
  7. Experimental Setting
  8. Overall Results
  9. Ablation Study
  10. Post-edited Model Analysis
  11. Conclusion

Figures (2)

  • Figure 1: Overall structure of HORSE. $\Delta \theta$ denotes the weight update, $R$ denotes the residual matrix, which is the key variable used to generate $\Delta \theta$, and $L$ denotes the loss for training the hypernetwork. Details of the HyperNetwork are provided in Section \ref{['HyperNetwork']}.
  • Figure 2: Performance under different numbers of edits.