Is Bigger Edit Batch Size Always Better? -- An Empirical Study on Model Editing with Llama-3

TL;DR

This paper investigates knowledge editing for large language models by evaluating three locate-and-edit methods—ROME, MEMIT, and EMMET—on Llama-3 under a preservation-memorization objective. It systematically compares sequential editing, batch editing, and sequential-batch editing across up to 4096 edits, revealing that larger batch edits often harm performance more than many small, sequential edits. The findings show that the earliest layers (e.g., layer 1) can be optimal for edits in Llama-3 and that a sequential-batched approach with a batch size around 1024 yields better scalability than pure batching. Overall, the work challenges the assumption that bigger batch sizes improve editing and suggests combining sequential and batched strategies for efficient, scalable knowledge updates in transformers.

Abstract

This study presents a targeted model editing analysis focused on the latest large language model, Llama-3. We explore the efficacy of popular model editing techniques - ROME, MEMIT, and EMMET, which are designed for precise layer interventions. We identify the most effective layers for targeted edits through an evaluation that encompasses up to 4096 edits across three distinct strategies: sequential editing, batch editing, and a hybrid approach we call as sequential-batch editing. Our findings indicate that increasing edit batch-sizes may degrade model performance more significantly than using smaller edit batches sequentially for equal number of edits. With this, we argue that sequential model editing is an important component for scaling model editing methods and future research should focus on methods that combine both batched and sequential editing. This observation suggests a potential limitation in current model editing methods which push towards bigger edit batch sizes, and we hope it paves way for future investigations into optimizing batch sizes and model editing performance.

Figures (6)

• Figure 1: Post-edit performance of various metrics for Llama3-8b model using MEMIT and ROME on various layers. To make the above plots, each layer is edited with 1000 facts, one at a time and non-sequentially.
• Figure 2: Figure shows various metric results (PS, NS, ES, S) after a batch MEMIT (16, 64, 256, 1024, 4096) edit.
• Figure 3: Figure shows various metric results (PS, NS, ES, S) after a batch EMMET (16, 64, 256, 1024, 4096) edit.
• Figure 4: Single layer sequential editing performance of MEMIT for various batch sizes.
• Figure 5: Single layer sequential editing performance of EMMET for various batch sizes