Knowledge Editing on Black-box Large Language Models

TL;DR

A novel postEdit framework is introduced to tackle privacy leaks of editing data and style over-editing in current methods, resolving privacy concerns through downstream post-processing and maintaining textual style consistency via fine-grained editing to original responses.

Abstract

Knowledge editing (KE) aims to efficiently and precisely modify the behavior of large language models (LLMs) to update specific knowledge without negatively influencing other knowledge. Current research primarily focuses on white-box LLMs editing, overlooking an important scenario: black-box LLMs editing, where LLMs are accessed through interfaces and only textual output is available. In this paper, we first officially introduce KE on black-box LLMs and then propose a comprehensive evaluation framework to overcome the limitations of existing evaluations that are not applicable to black-box LLMs editing and lack comprehensiveness. To tackle privacy leaks of editing data and style over-editing in current methods, we introduce a novel postEdit framework, resolving privacy concerns through downstream post-processing and maintaining textual style consistency via fine-grained editing to original responses. Experiments and analysis on two benchmarks demonstrate that postEdit outperforms all baselines and achieves strong generalization, especially with huge improvements on style retention (average $+20.82\%\uparrow$).

Figures (6)

• Figure 1: Illustration of Knowledge Editing and comparison of two editing scenarios, where black-box LLMs editing constrains LLMs to only obtain textual output.
• Figure 2: Comparison of different KE frameworks for black-box LLM editing. IKE operates on LLM input, and SERAC performs editing using a surrogate model parallel to LLM, while our postEdit edits after the output of LLM and achieves both privacy protection and style retention.
• Figure 3: The overall architecture of postEdit. The post-editor is trained to learn: (1) distinguish between INS and OOS queries; (2) edit the output of INS queries while preserving style. Pseudo-code is provided in Appendix \ref{['subapp:postedit']}.
• Figure 4: Performance under different base LLMs and datasets, where Editing Score is the average of TE and SE, and Retention Score is the average of TR and SR.
• Figure 5: Performance curves of the post-editor at different scales on CounterFact.