Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Modalities Remember: Continual Learning for Multimodal Knowledge Graphs

Linyu Li, Zhi Jin, Yichi Zhang, Dongming Jin, Yuanpeng He, Haoran Duan, Gadeng Luosang, Nyima Tashi

Abstract

Real-world multimodal knowledge graphs (MMKGs) are dynamic, with new entities, relations, and multimodal knowledge emerging over time. Existing continual knowledge graph reasoning (CKGR) methods focus on structural triples and cannot fully exploit multimodal signals from new entities. Existing multimodal knowledge graph reasoning (MMKGR) methods, however, usually assume static graphs and suffer catastrophic forgetting as graphs evolve. To address this gap, we present a systematic study of continual multimodal knowledge graph reasoning (CMMKGR). We construct several continual multimodal knowledge graph benchmarks from existing MMKG datasets and propose MRCKG, a new CMMKGR model. Specifically, MRCKG employs a multimodal-structural collaborative curriculum to schedule progressive learning based on the structural connectivity of new triples to the historical graph and their multimodal compatibility. It also introduces a cross-modal knowledge preservation mechanism to mitigate forgetting through entity representation stability, relational semantic consistency, and modality anchoring. In addition, a multimodal contrastive replay scheme with a two-stage optimization strategy reinforces learned knowledge via multimodal importance sampling and representation alignment. Experiments on multiple datasets show that MRCKG preserves previously learned multimodal knowledge while substantially improving the learning of new knowledge.

When Modalities Remember: Continual Learning for Multimodal Knowledge Graphs

Abstract

Real-world multimodal knowledge graphs (MMKGs) are dynamic, with new entities, relations, and multimodal knowledge emerging over time. Existing continual knowledge graph reasoning (CKGR) methods focus on structural triples and cannot fully exploit multimodal signals from new entities. Existing multimodal knowledge graph reasoning (MMKGR) methods, however, usually assume static graphs and suffer catastrophic forgetting as graphs evolve. To address this gap, we present a systematic study of continual multimodal knowledge graph reasoning (CMMKGR). We construct several continual multimodal knowledge graph benchmarks from existing MMKG datasets and propose MRCKG, a new CMMKGR model. Specifically, MRCKG employs a multimodal-structural collaborative curriculum to schedule progressive learning based on the structural connectivity of new triples to the historical graph and their multimodal compatibility. It also introduces a cross-modal knowledge preservation mechanism to mitigate forgetting through entity representation stability, relational semantic consistency, and modality anchoring. In addition, a multimodal contrastive replay scheme with a two-stage optimization strategy reinforces learned knowledge via multimodal importance sampling and representation alignment. Experiments on multiple datasets show that MRCKG preserves previously learned multimodal knowledge while substantially improving the learning of new knowledge.

Paper Structure

This paper contains 28 sections, 18 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Knowledge Graph Reasoning
  4. Continual Learning for Knowledge Graph Reasoning
  5. Preliminaries and Problem Definition
  6. Method
  7. Framework Overview
  8. Scalable Multimodal Encoding Backbone
  9. Multimodal Structure-aware Curriculum Learning (MSCL)
  10. Curriculum Score
  11. Progressive Training Procedure
  12. Cross-Modality Knowledge Preservation (CMKP)
  13. Entity-Level Stability
  14. Consistency of Relational Semantic Patterns
  15. Modal Anchoring
  16. ...and 13 more sections

Figures (5)

  • Figure 1: CMMKG stores knowledge in the form of triplets; however, unlike traditional knowledge graphs, its multimodal information also continuously evolves over time.
  • Figure 2: Overall framework of MRCKG for continual multimodal knowledge graph reasoning.
  • Figure 3: Analysis on DB15K-Entity. (a) Per-snapshot MRR on new vs. old triples; (b) $\mathcal{S}_0$ forgetting curves; (c) MRR--BWT Pareto front; (d) multi-metric radar plot.
  • Figure 4: Per-snapshot MRR bar plots of MRCKG on the three DB15K settings.
  • Figure 5: (a) Error type distribution of MRCKG; (b) Hits@1 of three methods across snapshots.