Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models

Qizhou Chen, Dakan Wang, Taolin Zhang, Zaoming Yan, Chengsong You, Chengyu Wang, Xiaofeng He

TL;DR

UniEdit addresses the lack of open-domain knowledge editing benchmarks by constructing a Wikidata-based dataset and unifying evaluation of generality and locality through Neighborhood Multi-hop Chain Sampling (NMCS). The pipeline converts structured knowledge into natural language prompts via Deepseek-V3, enabling controlled multi-hop ripple-effect testing across 25 domains. Experiments across multiple LLM backbones and editors reveal strong improvements in reliability after edits but persistent challenges in generality, with retrieval-based and in-context approaches offering the best generalization. Overall, UniEdit provides a scalable, open-domain toolkit and benchmark to drive the development and fair comparison of LLM editors, guiding future work toward robust, domain-spanning knowledge updates.

Abstract

Model editing aims to enhance the accuracy and reliability of large language models (LLMs) by efficiently adjusting their internal parameters. Currently, most LLM editing datasets are confined to narrow knowledge domains and cover a limited range of editing evaluation. They often overlook the broad scope of editing demands and the diversity of ripple effects resulting from edits. In this context, we introduce UniEdit, a unified benchmark for LLM editing grounded in open-domain knowledge. First, we construct editing samples by selecting entities from 25 common domains across five major categories, utilizing the extensive triple knowledge available in open-domain knowledge graphs to ensure comprehensive coverage of the knowledge domains. To address the issues of generality and locality in editing, we design an Neighborhood Multi-hop Chain Sampling (NMCS) algorithm to sample subgraphs based on a given knowledge piece to entail comprehensive ripple effects to evaluate. Finally, we employ proprietary LLMs to convert the sampled knowledge subgraphs into natural language text, guaranteeing grammatical accuracy and syntactical diversity. Extensive statistical analysis confirms the scale, comprehensiveness, and diversity of our UniEdit benchmark. We conduct comprehensive experiments across multiple LLMs and editors, analyzing their performance to highlight strengths and weaknesses in editing across open knowledge domains and various evaluation criteria, thereby offering valuable insights for future research endeavors.

UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models

TL;DR

UniEdit addresses the lack of open-domain knowledge editing benchmarks by constructing a Wikidata-based dataset and unifying evaluation of generality and locality through Neighborhood Multi-hop Chain Sampling (NMCS). The pipeline converts structured knowledge into natural language prompts via Deepseek-V3, enabling controlled multi-hop ripple-effect testing across 25 domains. Experiments across multiple LLM backbones and editors reveal strong improvements in reliability after edits but persistent challenges in generality, with retrieval-based and in-context approaches offering the best generalization. Overall, UniEdit provides a scalable, open-domain toolkit and benchmark to drive the development and fair comparison of LLM editors, guiding future work toward robust, domain-spanning knowledge updates.

Abstract

Model editing aims to enhance the accuracy and reliability of large language models (LLMs) by efficiently adjusting their internal parameters. Currently, most LLM editing datasets are confined to narrow knowledge domains and cover a limited range of editing evaluation. They often overlook the broad scope of editing demands and the diversity of ripple effects resulting from edits. In this context, we introduce UniEdit, a unified benchmark for LLM editing grounded in open-domain knowledge. First, we construct editing samples by selecting entities from 25 common domains across five major categories, utilizing the extensive triple knowledge available in open-domain knowledge graphs to ensure comprehensive coverage of the knowledge domains. To address the issues of generality and locality in editing, we design an Neighborhood Multi-hop Chain Sampling (NMCS) algorithm to sample subgraphs based on a given knowledge piece to entail comprehensive ripple effects to evaluate. Finally, we employ proprietary LLMs to convert the sampled knowledge subgraphs into natural language text, guaranteeing grammatical accuracy and syntactical diversity. Extensive statistical analysis confirms the scale, comprehensiveness, and diversity of our UniEdit benchmark. We conduct comprehensive experiments across multiple LLMs and editors, analyzing their performance to highlight strengths and weaknesses in editing across open knowledge domains and various evaluation criteria, thereby offering valuable insights for future research endeavors.
Paper Structure (31 sections, 3 equations, 10 figures, 14 tables, 1 algorithm)

This paper contains 31 sections, 3 equations, 10 figures, 14 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Knowledge Editing Methods
  4. Knowledge Editing Benchmarks
  5. Problem Definition
  6. UniEdit
  7. Data Construction Process
  8. Dataset Statistics
  9. Experiments
  10. Experiments Settings
  11. Overall Performance
  12. Performance Across Domains
  13. Performance Across Evaluation Criteria
  14. Domain Generalization of Edit Training
  15. Conclusion, Limitation and Future Works
  16. ...and 16 more sections

Figures (10)

  • Figure 1: Data composition of UniEdit, covering up to 25 different domains extracted in Wikidata. Given an editing triple (highlighted with a red edge), generality and locality structures are sampled as multi-hop chain subgraphs from its neighborhood. The generality subgraphs include the entire editing triple, while locality refers to all other cases. In each subgraph, a node is selected to serve as the cloze target, forming a single chain prompt if it is an endpoint, or a double chain prompt otherwise (only single chain shown for locality here). Beyond prompt structural differences, locality samples further classified into six types according to their cross-features with the editing triple (see Appendix \ref{['sec_loc_structure_cor_criteria']} for correspondence with the criteria).
  • Figure 2: Data construction pipeline of UniEdit. Steps 1–3 include data preprocessing, domain-specific entity retrieval, and sampling of relevant triples based on the domain entity. In Step 4, generality and locality QA chains are sampled using NMCS algorithm. In Step 5, the final data is generated based on the sampled QA chains, where F and B indicate the forward and backward directions, respectively—referring to the prompt generation direction with respect to the triple.
  • Figure 3: Data count statistics of UniEdit across: (a) domains, (b) multi-hop counts and query chain structures (G., L., S., and D. represent generality, locality, single, and double, respectively), and (d, e) the top 15 combinations of recognized evaluation criteria. (c) displays the frequency statistics of nouns in entity descriptions.
  • Figure 4: Editing performance on UniEdit across domains, with each metric representing the average result across three post-edit backbones. The color bands (top to bottom) indicate reliability (green), generality (blue), and locality (red), with ranges normalized across domains (rows).
  • Figure 5: Editing performance across combinations of generality and locality evaluation criteria. The left half of each radar chart shows the evaluation results for a single criterion, while the symmetrical right half reflects the results after combining it with others.
  • ...and 5 more figures