Learning with Dual-level Noisy Correspondence for Multi-modal Entity Alignment

TL;DR

This work tackles the practical problem of dual-level noisy correspondence (DNC) in multi-modal entity alignment (MMEA) by introducing RULE, a reliability-guided framework. RULE estimates intra-entity and inter-graph reliability using uncertainty and consensus, partitions cross-graph pairs into clean and noisy groups, and applies robust intra-entity fusion and inter-graph discrepancy elimination. A test-time correspondence reasoning (TTR) module using a multimodal LLM enhances inference by uncovering latent attribute-attribute connections, yielding improved cross-graph entity identification. Extensive experiments on five benchmarks, including scenarios with artificially injected noise, demonstrate that RULE robustly outperforms seven state-of-the-art methods and offers strong, practical improvements for real-world MMKG alignment.

Abstract

Multi-modal entity alignment (MMEA) aims to identify equivalent entities across heterogeneous multi-modal knowledge graphs (MMKGs), where each entity is described by attributes from various modalities. Existing methods typically assume that both intra-entity and inter-graph correspondences are faultless, which is often violated in real-world MMKGs due to the reliance on expert annotations. In this paper, we reveal and study a highly practical yet under-explored problem in MMEA, termed Dual-level Noisy Correspondence (DNC). DNC refers to misalignments in both intra-entity (entity-attribute) and inter-graph (entity-entity and attribute-attribute) correspondences. To address the DNC problem, we propose a robust MMEA framework termed RULE. RULE first estimates the reliability of both intra-entity and inter-graph correspondences via a dedicated two-fold principle. Leveraging the estimated reliabilities, RULE mitigates the negative impact of intra-entity noise during attribute fusion and prevents overfitting to noisy inter-graph correspondences during inter-graph discrepancy elimination. Beyond the training-time designs, RULE further incorporates a correspondence reasoning module that uncovers the underlying attribute-attribute connection across graphs, guaranteeing more accurate equivalent entity identification. Extensive experiments on five benchmarks verify the effectiveness of our method against the DNC compared with seven state-of-the-art methods.The code is available at \href{https://github.com/XLearning-SCU/RULE}{XLearning-SCU/RULE}

Figures (12)

• Figure 1: (a) Dual-level Noisy Correspondence occurs in both intra-entity level (i.e., entity-attribute pairs such as ($\tilde{x}_i,\tilde{x}_i^m$)) and the inter-graph level (i.e., entity-entity ($x_{j},\tilde{x}_{k}$) or attribute-attribute pairs ($x_{j}^{m},\tilde{x}_{k}^{m}$)). (b) Observations: On the one hand, both vanilla adaptive fusion (AF) and concatenation (Concat) tend to integrate erroneous attributes and thus degrade performance, while our method achieves reliable fusion against inter-graph NC. On the other hand, existing methods suffer in cross-graph alignment when encountering inter-graph NC, whereas our method boosts performance by mitigating the negative impact of intra-entity NC. (c) Misidentification at Test Time: Seemingly similar attribute pairs may prevent the query entity from being associated with its equivalent entity. For instance, the implicit connection between the football player "Cristiano Ronaldo" and his home country are often overlooked, resulting in misidentification during entity alignment.
• Figure 2: Overview of our method RULE. Given two MMKGs $\mathcal{G}$ and $\tilde{\mathcal{G}}$, RULE first projects the entity attributes into a shared latent space and computes cross-graph attribute similarities. These similarities are used to estimate the reliability of inter-graph correspondences and categorize cross-graph pairs into three subsets: $\mathcal{S}_C$, $\mathcal{S}_I$, and $\mathcal{S}_U$. Subsequently, the robust intra-entity attribute fusion module and robust inter-graph discrepancy elimination are employed to mitigate the impact of both intra-entity and inter-graph noisy correspondences. Beyond training-time robustness, a test-time correspondence reasoning module uncovers latent attribute-attribute connections across graphs, enabling more accurate equivalent entity identification during inference.
• Figure 3: Analytic studies of various DNC ratios and reliability in Eq. \ref{['eq: reliablity_weights']}.
• Figure 4: Quantitative analysis of the uncertainty and consensus on the name attribute.
• Figure 5: Visualization of the reliability for clean entity pairs and those with manually injected E-A NC in the image and name attributes. From top to bottom, the three rows denote the structure, image, and name attributes, respectively.

Theorems & Definitions (8)

• Definition 1
• Theorem 1
• Definition 2
• Definition 3
• Theorem 2
• proof
• Theorem 3
• proof