Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Out-of-Distribution Generalization via Invariant Trajectories for Multimodal Large Language Model Editing

Jiajie Su, Haoyuan Wang, Xiaohua Feng, Yunshan Ma, Xiaobo Xia, Yuyuan Li, Xiaolin Zheng, Jianmao Xiao, Chaochao Chen

TL;DR

This work reframes multimodal LLM editing as an out-of-distribution generalization problem, targeting invariant causal trajectories across cross-modal prompts. It introduces ODEdit, a plug-and-play framework that combines a tripartite OOD risk (reliability, locality, generality) with Edit Trajectory Invariant Learning (ETIL) and a total variation penalty to stabilize edit trajectories. The approach leverages IRM principles and a TV-$\ell_1$ regularizer to achieve robust, cross-environment knowledge edits across unimodal and multimodal backbones, validated on MMEdit benchmarks with consistent gains in reliability, locality, and generality. Empirical results show improved one-step and long-term editing performance and robustness to semantic-neighboring and out-of-distribution prompts, underscoring the method's practical potential for reliable MLLM knowledge updates.

Abstract

Knowledge editing emerges as a crucial technique for efficiently correcting incorrect or outdated knowledge in large language models (LLM). Existing editing methods for unimodal LLM rely on a rigid parameter-to-output mapping, which causes causal-underfit and causal-overfit in cascaded reasoning for Multimodal LLM (MLLM). In this paper, we reformulate MLLM editing as an out-of-distribution (OOD) generalization problem, where the goal is to discern semantic shift with factual shift and thus achieve robust editing among diverse cross-modal prompting. The key challenge of this OOD problem lies in identifying invariant causal trajectories that generalize accurately while suppressing spurious correlations. To address it, we propose ODEdit, a plug-and-play invariant learning based framework that optimizes the tripartite OOD risk objective to simultaneously enhance editing reliability, locality, and generality.We further introduce an edit trajectory invariant learning method, which integrates a total variation penalty into the risk minimization objective to stabilize edit trajectories against environmental variations. Theoretical analysis and extensive experiments demonstrate the effectiveness of ODEdit.

Out-of-Distribution Generalization via Invariant Trajectories for Multimodal Large Language Model Editing

TL;DR

This work reframes multimodal LLM editing as an out-of-distribution generalization problem, targeting invariant causal trajectories across cross-modal prompts. It introduces ODEdit, a plug-and-play framework that combines a tripartite OOD risk (reliability, locality, generality) with Edit Trajectory Invariant Learning (ETIL) and a total variation penalty to stabilize edit trajectories. The approach leverages IRM principles and a TV- regularizer to achieve robust, cross-environment knowledge edits across unimodal and multimodal backbones, validated on MMEdit benchmarks with consistent gains in reliability, locality, and generality. Empirical results show improved one-step and long-term editing performance and robustness to semantic-neighboring and out-of-distribution prompts, underscoring the method's practical potential for reliable MLLM knowledge updates.

Abstract

Knowledge editing emerges as a crucial technique for efficiently correcting incorrect or outdated knowledge in large language models (LLM). Existing editing methods for unimodal LLM rely on a rigid parameter-to-output mapping, which causes causal-underfit and causal-overfit in cascaded reasoning for Multimodal LLM (MLLM). In this paper, we reformulate MLLM editing as an out-of-distribution (OOD) generalization problem, where the goal is to discern semantic shift with factual shift and thus achieve robust editing among diverse cross-modal prompting. The key challenge of this OOD problem lies in identifying invariant causal trajectories that generalize accurately while suppressing spurious correlations. To address it, we propose ODEdit, a plug-and-play invariant learning based framework that optimizes the tripartite OOD risk objective to simultaneously enhance editing reliability, locality, and generality.We further introduce an edit trajectory invariant learning method, which integrates a total variation penalty into the risk minimization objective to stabilize edit trajectories against environmental variations. Theoretical analysis and extensive experiments demonstrate the effectiveness of ODEdit.
Paper Structure (29 sections, 2 theorems, 53 equations, 5 figures, 6 tables)

This paper contains 29 sections, 2 theorems, 53 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Methodology
  5. Problem Setting
  6. Semantic-Factual Shift Disentanglement
  7. Edit Trajectory Invariant Learning
  8. Experiment
  9. Experimental Setup
  10. Performance on One-step Knowledge Editing
  11. Performance on Long-term Knowledge Editing
  12. In-Depth Analysis
  13. Conclusion and Future Work
  14. Proofs
  15. Equivalence between OOD-w and IRM Formulation
  16. ...and 14 more sections

Key Result

Proposition 4.1

Under the condition that the environment variability is channeled through the classifier $\omega$, it satisfies the identity $\mathcal{R}_{\text{edit}}(\phi_e, e) \equiv \mathcal{R}_{\text{edit}}(\omega(e) \circ \phi_e)$. The OOD editing objective in Eq.(eq:ood) admits the following equivalent IRM f

Figures (5)

  • Figure 1: The motivation of ODEdit. The left presents why previous editing work targeted at unimodal LLM is not effective in MLLM. The right denotes two shifts in this editing OOD problem.
  • Figure 2: Ablation studies of IRM-TV optimization.
  • Figure 3: The t-SNE distributions of the latent representations on original prompts (SRC) and rephrase prompts (GEN) in MLLM. The curves depict the marginal distributions along the two dimensions, with $\beta_x$ and $\beta_y$ representing the proportion of the overlap. (a)-(c) denotes the MEND, (d)-(f) denotes the MEND+ODEdit.
  • Figure 4: Effects of learning rate and layer depth.
  • Figure 5: Case studies on the evaluation for generality, image locality, and text locality.

Theorems & Definitions (6)

  • Proposition 4.1: Equivalence between OOD-$\omega$ and IRM
  • proof
  • Proposition 4.2: IRM-TV objective Achieves Editing OOD with a varying $\lambda$
  • proof
  • Definition 3.1: Semantic Shift
  • Definition 3.2: Factual Shift