Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReasonEdit: Editing Vision-Language Models using Human Reasoning

Jiaxing Qiu, Kaihua Hou, Roxana Daneshjou, Ahmed Alaa, Thomas Hartvigsen

TL;DR

This work tackles the challenge of editing vision–language models on reasoning-heavy tasks by enabling human reasoning to guide updates without altering unrelated behavior. It introduces ReasonEdit, a retrieval-based editor that stores reasoning as a multimodal codebook of image–text entries and uses visual-evidence patches with topology-balanced embeddings to retrieve relevant facts during inference. Empirical results across four VLMs and two rationale-VQA datasets show state-of-the-art editing performance and novel generalization forms, such as rationale-gen and chain-of-error-gen, while maintaining efficiency in sequential editing. The approach offers practical, real-time editing capabilities and provides insights into cross-modal embedding topology that inform embedding selection for robust retrieval-based edits.

Abstract

Model editing aims to correct errors in large, pretrained models without altering unrelated behaviors. While some recent works have edited vision-language models (VLMs), no existing editors tackle reasoning-heavy tasks, which typically require humans and models to reason about images.We therefore propose ReasonEdit, the first VLM editor to let users explain their reasoning during editing, introducing a new, practical model editing setup. ReasonEdit continuously stores human reasoning in a codebook, and retrieves only relevant facts during inference using a novel topology-balanced multimodal embedding method inspired by network science. Across four VLMs on multiple rationale-based visual question answering datasets, ReasonEdit achieves state-of-the-art editing performance, ultimately showing that using human reasoning during editing greatly improves edit generalization.

ReasonEdit: Editing Vision-Language Models using Human Reasoning

TL;DR

This work tackles the challenge of editing vision–language models on reasoning-heavy tasks by enabling human reasoning to guide updates without altering unrelated behavior. It introduces ReasonEdit, a retrieval-based editor that stores reasoning as a multimodal codebook of image–text entries and uses visual-evidence patches with topology-balanced embeddings to retrieve relevant facts during inference. Empirical results across four VLMs and two rationale-VQA datasets show state-of-the-art editing performance and novel generalization forms, such as rationale-gen and chain-of-error-gen, while maintaining efficiency in sequential editing. The approach offers practical, real-time editing capabilities and provides insights into cross-modal embedding topology that inform embedding selection for robust retrieval-based edits.

Abstract

Model editing aims to correct errors in large, pretrained models without altering unrelated behaviors. While some recent works have edited vision-language models (VLMs), no existing editors tackle reasoning-heavy tasks, which typically require humans and models to reason about images.We therefore propose ReasonEdit, the first VLM editor to let users explain their reasoning during editing, introducing a new, practical model editing setup. ReasonEdit continuously stores human reasoning in a codebook, and retrieves only relevant facts during inference using a novel topology-balanced multimodal embedding method inspired by network science. Across four VLMs on multiple rationale-based visual question answering datasets, ReasonEdit achieves state-of-the-art editing performance, ultimately showing that using human reasoning during editing greatly improves edit generalization.
Paper Structure (47 sections, 20 equations, 13 figures, 2 tables, 1 algorithm)

This paper contains 47 sections, 20 equations, 13 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning-Enhanced VLM Editing
  4. Problem Definition
  5. ReasonEdit
  6. Topology-Aware Multimodal Embedding
  7. Experiments
  8. Evaluation Metrics
  9. Experimental Setup
  10. Comparisons to Prior Editors
  11. Sequential Editing and Computing Efficiency
  12. Ablation Studies
  13. Topology-Aware Embedding Method Selection
  14. Conclusion
  15. Topology-Aware Multimodal Embedding
  16. ...and 32 more sections

Figures (13)

  • Figure 1: Reasoning-enhanced VLM editing lets users provide detailed feedback and reasoning, enabling broader generalization.
  • Figure 2: ReasonEdit allows users to provide detailed feedback when VLMs make errors in reasoning-heavy tasks. It converts each edit into image–text entries by pairing textual details from human reasoning with visual evidence image patches, and stores them in a codebook using a topology-aware multimodal embedding. At inference, it retrieves most relevant facts as a new query's context.
  • Figure 3: Visualization of embedding networks. Each color represents one image (and its augmentations) paired with varying texts. Vision-biased embeddings by a single vision layer cluster irrelevant texts due to image similarity. Language-biased embeddings produced by a language layer cluster different images due to text similarity. Such unimodal-biases lead to false distance-based retrievals. In contrast, The topology-balanced embedding aligns with joint image–text similarity. Demonstration with InstructBLIP-Vicuna-7B.
  • Figure 4: Sequential editing performance and efficiency across editors. ReasonEdit achieves the best sample generalities, rationale-informed generalities, as well as high reliability and locality. Trajectories are smoothed using moving average with a 5-step window.
  • Figure 5: Results for ablation studies.
  • ...and 8 more figures

Theorems & Definitions (4)

  • Definition 1.1: Expected Unimodal Partition
  • Definition 1.2: Expected Bimodal Partition
  • Definition 1.3: Vision Bias
  • Definition 1.4: Language Bias