Aligning Language Models with Real-time Knowledge Editing
Chenming Tang, Yutong Yang, Kexue Wang, Yunfang Wu
TL;DR
This work tackles the mismatch between static knowledge-edit benchmarks and the real-time evolution of factual knowledge by introducing CRAFT, an ever-updating Chinese knowledge-editing benchmark focused on statistics and finance. It reveals limitations of existing editing methods under real-time dynamics and presents KEDAS, a retrieval-based alignment framework that uses diverse edit augmentation, a filter-driven memory retriever, and self-adaptive inference to improve edit success, locality, and portability. Empirical results on CRAFT show KEDAS achieving state-of-the-art performance across most metrics, demonstrating strong real-time adaptability and robustness. The dataset and framework provide a practical path toward reliable, interpretable, and scalable knowledge editing in production LLM systems.
Abstract
Knowledge editing aims to modify outdated knowledge in large language models (LLMs) efficiently while retaining their original capabilities. Mainstream benchmarks for knowledge editing are predominantly static and fail to keep in pace with the evolving real-world knowledge. In this work, we introduce CRAFT, an ever-evolving real-world benchmark for knowledge editing. It features well-designed paired edits for composite reasoning, and evaluates models on alias portability as well as temporal and common-sense locality, making it a challenging knowledge editing benchmark on which previous knowledge editing methods hardly achieve balanced performance. Towards flexible real-time editing, we propose KEDAS, a novel paradigm of knowledge editing alignment featuring diverse edit augmentation and self-adaptive post-alignment inference, which exhibits significant performance gain on CRAFT compared to previous methods. All of our code and data are available at https://anonymous.4open.science/r/CRAFT-KEDAS.