Mitigating Modality Bias in Multi-modal Entity Alignment from a Causal Perspective

TL;DR

This work tackles visual modality bias in Multi-Modal Entity Alignment (MMEA) by introducing CDMEA, a counterfactual debiasing framework grounded in causal learning. By modeling a four-variable causal graph with visual ($V$), graph ($G$), fused ($M$) modalities and the prediction outcome ($Y$), the method derives the Total Indirect Effect ($TIE$) from the Total Effect ($TE$) and Natural Direct Effect ($NDE$), using a tunable parameter $\beta$ to control the direct visual influence. CDMEA employs dedicated Visual, Graph, and Fused modality encoders, a prediction fusion strategy, and InfoNCE-based training losses, enabling effective cross-modal fusion while suppressing image-based shortcuts during inference. Across 9 benchmarks (cross-KG and bilingual), CDMEA outperforms 14 SOTA baselines, with pronounced gains in low-similarity images, high-noise, and low-resource scenarios, demonstrating robust and scalable debiasing in MMEA. The approach offers practical impact for improving reliable entity alignment in noisy, resource-constrained, and heterogeneous MMKG environments. $TE$, $NDE$, and $TIE$ are central to the debiasing strategy, ensuring that the model leverages both visual and graph cues without allow­ing visual shortcuts to dominate predictions.

Abstract

Multi-Modal Entity Alignment (MMEA) aims to retrieve equivalent entities from different Multi-Modal Knowledge Graphs (MMKGs), a critical information retrieval task. Existing studies have explored various fusion paradigms and consistency constraints to improve the alignment of equivalent entities, while overlooking that the visual modality may not always contribute positively. Empirically, entities with low-similarity images usually generate unsatisfactory performance, highlighting the limitation of overly relying on visual features. We believe the model can be biased toward the visual modality, leading to a shortcut image-matching task. To address this, we propose a counterfactual debiasing framework for MMEA, termed CDMEA, which investigates visual modality bias from a causal perspective. Our approach aims to leverage both visual and graph modalities to enhance MMEA while suppressing the direct causal effect of the visual modality on model predictions. By estimating the Total Effect (TE) of both modalities and excluding the Natural Direct Effect (NDE) of the visual modality, we ensure that the model predicts based on the Total Indirect Effect (TIE), effectively utilizing both modalities and reducing visual modality bias. Extensive experiments on 9 benchmark datasets show that CDMEA outperforms 14 state-of-the-art methods, especially in low-similarity, high-noise, and low-resource data scenarios.

Figures (10)

• Figure 1: Examples of MMEA between two MMKGs: (a) entity equivalence identification, (b) MMEA with supportive visual cues, and (c) MMEA with negative visual cues.
• Figure 2: Experimental results on equivalent entity pairs with low-similarity images from the FB-DB15K dataset.
• Figure 3: Example of the causal graph. (a) Factual world. (b, c) Counterfactual world. White nodes represent observed variables, while blue-striped nodes are counterfactual variables.
• Figure 4: (a) Causal graph for MMEA. (b) Counterfactual analysis illustrates the difference between factual and counterfactual inference outcomes for entities with observed values.
• Figure 5: The framework of the proposed CDMEA in the training phase and debiasing inference phase.