Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diffusion-based Hierarchical Negative Sampling for Multimodal Knowledge Graph Completion

Guanglin Niu, Xiaowei Zhang

TL;DR

This work tackles incomplete multimodal knowledge graphs by addressing the negative sampling bottleneck in MMKGC. It introduces diffusion-based hierarchical embedding generation (DiffHEG) to directly synthesize multimodal negative triples at multiple hardness levels, and a negative triple-adaptive training (NTAT) strategy with a hardness-adaptive loss (HAL) to adapt margins during learning. Empirical results on three MMKGC benchmarks show that DHNS, especially when paired with RotatE, outperforms state-of-the-art unimodal and multimodal models as well as existing negative sampling methods, demonstrating robust gains across diverse datasets. The framework provides a pluggable NS module and training paradigm, with public code available, enabling broader adoption for improved MMKGC model training.

Abstract

Multimodal Knowledge Graph Completion (MMKGC) aims to address the critical issue of missing knowledge in multimodal knowledge graphs (MMKGs) for their better applications. However, both the previous MMGKC and negative sampling (NS) approaches ignore the employment of multimodal information to generate diverse and high-quality negative triples from various semantic levels and hardness levels, thereby limiting the effectiveness of training MMKGC models. Thus, we propose a novel Diffusion-based Hierarchical Negative Sampling (DHNS) scheme tailored for MMKGC tasks, which tackles the challenge of generating high-quality negative triples by leveraging a Diffusion-based Hierarchical Embedding Generation (DiffHEG) that progressively conditions on entities and relations as well as multimodal semantics. Furthermore, we develop a Negative Triple-Adaptive Training (NTAT) strategy that dynamically adjusts training margins associated with the hardness level of the synthesized negative triples, facilitating a more robust and effective learning procedure to distinguish between positive and negative triples. Extensive experiments on three MMKGC benchmark datasets demonstrate that our framework outperforms several state-of-the-art MMKGC models and negative sampling techniques, illustrating the effectiveness of our DHNS for training MMKGC models. The source codes and datasets of this paper are available at https://github.com/ngl567/DHNS.

Diffusion-based Hierarchical Negative Sampling for Multimodal Knowledge Graph Completion

TL;DR

This work tackles incomplete multimodal knowledge graphs by addressing the negative sampling bottleneck in MMKGC. It introduces diffusion-based hierarchical embedding generation (DiffHEG) to directly synthesize multimodal negative triples at multiple hardness levels, and a negative triple-adaptive training (NTAT) strategy with a hardness-adaptive loss (HAL) to adapt margins during learning. Empirical results on three MMKGC benchmarks show that DHNS, especially when paired with RotatE, outperforms state-of-the-art unimodal and multimodal models as well as existing negative sampling methods, demonstrating robust gains across diverse datasets. The framework provides a pluggable NS module and training paradigm, with public code available, enabling broader adoption for improved MMKGC model training.

Abstract

Multimodal Knowledge Graph Completion (MMKGC) aims to address the critical issue of missing knowledge in multimodal knowledge graphs (MMKGs) for their better applications. However, both the previous MMGKC and negative sampling (NS) approaches ignore the employment of multimodal information to generate diverse and high-quality negative triples from various semantic levels and hardness levels, thereby limiting the effectiveness of training MMKGC models. Thus, we propose a novel Diffusion-based Hierarchical Negative Sampling (DHNS) scheme tailored for MMKGC tasks, which tackles the challenge of generating high-quality negative triples by leveraging a Diffusion-based Hierarchical Embedding Generation (DiffHEG) that progressively conditions on entities and relations as well as multimodal semantics. Furthermore, we develop a Negative Triple-Adaptive Training (NTAT) strategy that dynamically adjusts training margins associated with the hardness level of the synthesized negative triples, facilitating a more robust and effective learning procedure to distinguish between positive and negative triples. Extensive experiments on three MMKGC benchmark datasets demonstrate that our framework outperforms several state-of-the-art MMKGC models and negative sampling techniques, illustrating the effectiveness of our DHNS for training MMKGC models. The source codes and datasets of this paper are available at https://github.com/ngl567/DHNS.
Paper Structure (21 sections, 14 equations, 1 figure, 5 tables, 1 algorithm)

This paper contains 21 sections, 14 equations, 1 figure, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Knowledge Graph Completion
  4. Negative Sampling in Knowledge Graph Embedding
  5. Diffusion Models
  6. Methodology
  7. Preliminaries and Problem Definition
  8. Diffusion-based Hierarchical Embedding Generation
  9. Forward Diffusion and Reverse Denoising Procedures.
  10. Multimodal Conditioning and Multiple Hardness-Level Denoising.
  11. Negative Triple-Adaptive Training
  12. Experiments
  13. Experiment Settings
  14. MMKGC Datasets.
  15. Baseline Models.
  16. ...and 6 more sections

Figures (1)

  • Figure 1: The whole framework of our DHNS. MHLD means multiple hardness-level denoising. $\textbf{x}_{0:T}$ and $\hat{\textbf{x}}_{0:T}$ are the noised and the denoised embeddings in the range of time steps $[0, T]$ corresponding to an entity. $\textbf{x}_{T/20}^{struc}$, $\textbf{x}_{T/20}^{text}$ and $\textbf{x}_{T/20}^{vis}$ are three modality-specific (structural/textual/visual) denoised embeddings at the time step $T/20$. $\gamma_{T/20}$ denotes the margin adaptive to the negative triples with the hardness level $HL(\frac{T}{20})$.