An Exploration of Knowledge Editing for Arabic

TL;DR

This work is the first systematic study of knowledge editing in Arabic, addressing the scarcity of Arabic KE benchmarks and the challenges posed by morphology and diglossia. It evaluates four KE methods—ROME, MEMIT, ICE, and LTE—on Arabic translations of ZsRE and Counterfact, demonstrating that parameter-based approaches struggle with cross-lingual transfer, while instruction-tuned, multilingual LTE achieves robust edits across Arabic and other languages. By extending LTE to monolingual Arabic and bilingual Arabic-English training, the authors show improved editability and cross-lingual generalization, with multilingual training outperforming single-language variants. They also release Arabic KE benchmarks and multilingual LTE data to spur future research in low-resource and morphologically rich languages.

Abstract

While Knowledge Editing (KE) has been widely explored in English, its behavior in morphologically rich languages like Arabic remains underexamined. In this work, we present the first study of Arabic KE. We evaluate four methods (ROME, MEMIT, ICE, and LTE) on Arabic translations of the ZsRE and Counterfact benchmarks, analyzing both multilingual and cross-lingual settings. Our experiments on Llama-2-7B-chat show that parameter-based methods struggle with cross-lingual generalization, while instruction-tuned methods perform more robustly. We extend Learning-To-Edit (LTE) to a multilingual setting and show that joint Arabic-English training improves both editability and transfer. We release Arabic KE benchmarks and multilingual training for LTE data to support future research.

Figures (9)

• Figure 1: Comparison of ROME, MEMIT, and ICE on LLaMA2-7B-Chat across four metrics: reliability, generality, locality, and Portability
• Figure 2: Impact of the editing language on the reliability, generality, portability and locality metrics on the ZsRE and Counterfact datasets for Llama2-7B-Chat
• Figure 3: (a) Shows a comparison of the considered methods across the reliability, generality, locality, and portability metrics on the ZsRE dataset. (b) Shows a comparison of the averaged cross-lingual reliability scores on the ZsRE dataset and (c) Shows a comparison of the averaged cross-lingual generality scores on the ZsRE dataset. The x-axis in (b) and (c) refer to the language the edit is being applied in.
• Figure 4: Cross Lingual Reliability Metrics Comparison (ZsRE)
• Figure 5: Comparison of ROME, MEMIT, and ICE on LLaMA2-7B-Chat across four metrics: reliability, generality, locality, and Portability on the counterfact dataset