Progressively Modality Freezing for Multi-Modal Entity Alignment

TL;DR

The paper tackles multi-modal entity alignment by addressing alignment-irrelevant features and cross-modal inconsistencies. It introduces Progressive Modality Freezing (PMF), which encodes per-modality features, computes alignment-relevance scores, progressively freezes less useful features, and fuses remaining signals into a joint representation. A unified objective combining cross-modality association losses and cross-KG alignment losses guides training, enabling robust cross-modal and cross-graph alignment. Empirical results on nine datasets show state-of-the-art performance, validating the efficacy and efficiency of the progressive freezing strategy for robust MMKG alignment in diverse settings.

Abstract

Multi-Modal Entity Alignment aims to discover identical entities across heterogeneous knowledge graphs. While recent studies have delved into fusion paradigms to represent entities holistically, the elimination of features irrelevant to alignment and modal inconsistencies is overlooked, which are caused by inherent differences in multi-modal features. To address these challenges, we propose a novel strategy of progressive modality freezing, called PMF, that focuses on alignmentrelevant features and enhances multi-modal feature fusion. Notably, our approach introduces a pioneering cross-modal association loss to foster modal consistency. Empirical evaluations across nine datasets confirm PMF's superiority, demonstrating stateof-the-art performance and the rationale for freezing modalities. Our code is available at https://github.com/ninibymilk/PMF-MMEA.

Figures (8)

• Figure 1: Illustration of irrelevant vs. relevant features in multi-modal knowledge graphs.
• Figure 2: Overview of the PMF Model. The framework consists of three components: the Multi-Modal Entity Encoder (&\ref{['Multi-Modal Entity Encoder']}), Progressive Multi-Modality Feature Integration (&\ref{['Progressive Multi-Modality Feature Integration']}), and Cross-Graph Contrastive Learning (&\ref{['Multi-Modal Entity Alignment Objective']}). At epoch t, the multi-modal encoder ENC$_m$ transforms raw inputs from each modality graph into entity embeddings $\mathcal{H}^t_m$. Progressive Feature Integration (&\ref{['Progressive Features Integration']}) occurs during training, employing Irrelevant Feature Freezing (&\ref{['Irrelevant Feature Freezing']}) and performing Relevant Feature Fusion (&\ref{['Relevant Features Fusion']}) guided by Relevant Feature Measuring (&\ref{['Feature Relevance Measuring']}). The model is optimized using Cross-KG Alignment Loss $\mathcal{L}^t_{CKG}$ (&\ref{['Cross-KG Alignment Loss']}) and Cross-Modality Association Loss $\mathcal{L}^t_{CM}$ (&\ref{['Cross-Modality Association Loss']}).
• Figure 3: Comparison of variants of PMF on DBP15K in terms of ${\rm H}@1$.
• Figure 4: Impact of various modalities on DBP15K in terms of ${\rm H}@1$.
• Figure 5: Distribution of relevance scores across modalities for pre-aligned entity pairs in ${\rm DBP15K}_{\rm FR-EN}$.