Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ELMM: Efficient Lightweight Multimodal Large Language Models for Multimodal Knowledge Graph Completion

Wei Huang, Peining Li, Meiyu Liang, Xu Hou, Junping Du, Yingxia Shao, Guanhua Ye, Wu Liu, Kangkang Lu, Yang Yu

TL;DR

This work tackles the problem of completing multimodal knowledge graphs by leveraging large language models while addressing the inefficiencies caused by abundant image tokens. It introduces ELMM, which combines a Multi-view Visual Token Compressor (MVTC) for adaptive token compression, an attention pruning strategy with a linear compensation mechanism, and a multimodal knowledge reasoning completion layer to outperform state-of-the-art MKGC methods. Across FB15k-237-IMG and WN18-IMG, ELMM achieves superior accuracy (e.g., Hits@1/3/10) with substantially reduced inference time, demonstrating both accuracy and efficiency gains. The approach offers a practical pathway to scalable multimodal reasoning in MKGC and highlights the importance of targeted token selection and structural pruning in MLLMs for knowledge-intensive tasks.

Abstract

Multimodal Knowledge Graphs (MKGs) extend traditional knowledge graphs by incorporating visual and textual modalities, enabling richer and more expressive entity representations. However, existing MKGs often suffer from incompleteness, which hinder their effectiveness in downstream tasks. Therefore, multimodal knowledge graph completion (MKGC) task is receiving increasing attention. While large language models (LLMs) have shown promise for knowledge graph completion (KGC), their application to the multimodal setting remains underexplored. Moreover, applying Multimodal Large Language Models (MLLMs) to the task of MKGC introduces significant challenges: (1) the large number of image tokens per entity leads to semantic noise and modality conflicts, and (2) the high computational cost of processing large token inputs. To address these issues, we propose Efficient Lightweight Multimodal Large Language Models (ELMM) for MKGC. ELMM proposes a Multi-view Visual Token Compressor (MVTC) based on multi-head attention mechanism, which adaptively compresses image tokens from both textual and visual views, thereby effectively reducing redundancy while retaining necessary information and avoiding modality conflicts. Additionally, we design an attention pruning strategy to remove redundant attention layers from MLLMs, thereby significantly reducing the inference cost. We further introduce a linear projection to compensate for the performance degradation caused by pruning. Extensive experiments on benchmark FB15k-237-IMG and WN18-IMG demonstrate that ELMM achieves state-of-the-art performance while substantially improving computational efficiency, establishing a new paradigm for multimodal knowledge graph completion.

ELMM: Efficient Lightweight Multimodal Large Language Models for Multimodal Knowledge Graph Completion

TL;DR

This work tackles the problem of completing multimodal knowledge graphs by leveraging large language models while addressing the inefficiencies caused by abundant image tokens. It introduces ELMM, which combines a Multi-view Visual Token Compressor (MVTC) for adaptive token compression, an attention pruning strategy with a linear compensation mechanism, and a multimodal knowledge reasoning completion layer to outperform state-of-the-art MKGC methods. Across FB15k-237-IMG and WN18-IMG, ELMM achieves superior accuracy (e.g., Hits@1/3/10) with substantially reduced inference time, demonstrating both accuracy and efficiency gains. The approach offers a practical pathway to scalable multimodal reasoning in MKGC and highlights the importance of targeted token selection and structural pruning in MLLMs for knowledge-intensive tasks.

Abstract

Multimodal Knowledge Graphs (MKGs) extend traditional knowledge graphs by incorporating visual and textual modalities, enabling richer and more expressive entity representations. However, existing MKGs often suffer from incompleteness, which hinder their effectiveness in downstream tasks. Therefore, multimodal knowledge graph completion (MKGC) task is receiving increasing attention. While large language models (LLMs) have shown promise for knowledge graph completion (KGC), their application to the multimodal setting remains underexplored. Moreover, applying Multimodal Large Language Models (MLLMs) to the task of MKGC introduces significant challenges: (1) the large number of image tokens per entity leads to semantic noise and modality conflicts, and (2) the high computational cost of processing large token inputs. To address these issues, we propose Efficient Lightweight Multimodal Large Language Models (ELMM) for MKGC. ELMM proposes a Multi-view Visual Token Compressor (MVTC) based on multi-head attention mechanism, which adaptively compresses image tokens from both textual and visual views, thereby effectively reducing redundancy while retaining necessary information and avoiding modality conflicts. Additionally, we design an attention pruning strategy to remove redundant attention layers from MLLMs, thereby significantly reducing the inference cost. We further introduce a linear projection to compensate for the performance degradation caused by pruning. Extensive experiments on benchmark FB15k-237-IMG and WN18-IMG demonstrate that ELMM achieves state-of-the-art performance while substantially improving computational efficiency, establishing a new paradigm for multimodal knowledge graph completion.
Paper Structure (24 sections, 1 theorem, 15 equations, 4 figures, 5 tables)

This paper contains 24 sections, 1 theorem, 15 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Knowledge Graph Completion
  4. Multimodal Large Language Models
  5. Methodology
  6. Preliminaries
  7. Problem Formulation
  8. Multimodal Large Language Models
  9. ELMM Framework
  10. Multi-view Visual Token Compressor
  11. Attention Pruning Strategy
  12. Multimodal Knowledge Reasoning Completion Layer
  13. Optimization
  14. Experiments
  15. Datasets
  16. ...and 9 more sections

Key Result

Theorem 1

The initialization of $W_c$ that minimizes the expected compensation error satisfies: where $U\Sigma V^T$ is the SVD decomposition of $\mathbb{E}(x^{pruning})$, $\Sigma^{+}$ denotes the pseudoinverse of $\Sigma$.

Figures (4)

  • Figure 1: A description of each associated image region's contribution to the entity Taylor Swift in a different link.
  • Figure 2: Overview of the ELMM architecture. From left to right: (1) ELMM processes input text and images containing incomplete triplet information; (2) MVTC adaptively compresses image tokens from both textual and visual views; (3) MLLM attention pruning and compensation mechanism.
  • Figure 3: Cosine similarity between the input and output of the attention layer in the training data of FB15k-237-IMG and WN18-IMG. The calculation method is to take 1,000 data points from the training set, perform inference, and calculate the average cosine similarity.
  • Figure 4: Hyperparameter analysis results. Left: Relationship between Hits@1 and the number of attention pruning layers. Right: Relationship between Hits@1 and learning rate.

Theorems & Definitions (2)

  • Theorem 1
  • proof