KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Augmentations and Constraints

TL;DR

This work addresses the challenge of injecting new knowledge into large multimodal models while retaining prior knowledge to avoid catastrophic forgetting. It introduces Kore, a synergistic framework that combines knowledge-oriented augmentation (Kore-augmentation) to generate structured, multi-round dialogue and instruction data, with a covariance-based constraint (Kore-constraint) that initializes adapters in the null space of activation covariances to minimize interference with existing knowledge. The approach is validated across multiple LMMs and scales, showing superior performance in both knowledge adaptation and retention compared with full fine-tuning, LoRA, and continual-learning baselines, and it supports task-specific retention through knowledge-oriented constraints. The results demonstrate Kore’s universality, scalability, and potential for practical deployment in continual knowledge evolution of multimodal models.

Abstract

Large Multimodal Models encode extensive factual knowledge in their pre-trained weights. However, its knowledge remains static and limited, unable to keep pace with real-world developments, which hinders continuous knowledge acquisition. Effective knowledge injection thus becomes critical, involving two goals: knowledge adaptation (injecting new knowledge) and knowledge retention (preserving old knowledge). Existing methods often struggle to learn new knowledge and suffer from catastrophic forgetting. To address this, we propose KORE, a synergistic method of KnOwledge-oRientEd augmentations and constraints for injecting new knowledge into large multimodal models while preserving old knowledge. Unlike general text or image data augmentation, KORE automatically converts individual knowledge items into structured and comprehensive knowledge to ensure that the model accurately learns new knowledge, enabling accurate adaptation. Meanwhile, KORE stores previous knowledge in the covariance matrix of LMM's linear layer activations and initializes the adapter by projecting the original weights into the matrix's null space, defining a fine-tuning direction that minimizes interference with previous knowledge, enabling powerful retention. Extensive experiments on various LMMs, including LLaVA-v1.5-7B, LLaVA-v1.5-13B, and Qwen2.5-VL-7B, show that KORE achieves superior new knowledge injection performance and effectively mitigates catastrophic forgetting.

Figures (13)

• Figure 2: Overview of Kore, a synergistic method for knowledge-oriented augmentation and constraint.Kore-augmentation automatically converts each piece of knowledge into profound and structured knowledge. Kore-constraint minimizes interference with previous knowledge by initializing an adapter with null space that stores covariance matrix of previous knowledge.
• Figure 3: Comparison of Kore-augmentation(left) and general augmentation methods (right).
• Figure 4: Performance (higher is better) on (a) MME fu2023mme and (b) ScienceQA lu2022scienceqa after reconstruction. (c) Covariance matrix visualization for 4 different input activations in the 0-th block. We down-sample the heatmaps into 32×32. Similar patterns are marked in red circles.
• Figure 5: Comparison between Kore and baseline methods on fine-grained knowledge types.
• Figure 6: Performance of knowledge adaptation (K.A) and retention (K.R) under specific knowledge-oriented constraints.

Theorems & Definitions (2)

• Theorem 1
• Theorem 2