Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Jensen-Shannon Divergence Message-Passing for Rich-Text Graph Representation Learning

Zuo Wang, Ye Yuan

TL;DR

This work addresses representation learning in rich-text graphs by jointly modeling contextual and structural divergence alongside traditional similarity, via Jensen-Shannon Divergence Message-Passing (JSDMP). It introduces two Divergent Message-Passing GNNs, DMPGCN and DMPPRG, which compute divergence-aware edge weights to guide aggregation and feature fusion. Empirical results on six real-world datasets show that the proposed models outperform strong baselines, with notable gains in ACC and NMI across scientific articles, blogs, news, and Wikipedia graphs. The approach offers a principled way to leverage both content and structure in rich-text graphs and is extensible to larger language models for downstream NLP tasks.

Abstract

In this paper, we investigate how the widely existing contextual and structural divergence may influence the representation learning in rich-text graphs. To this end, we propose Jensen-Shannon Divergence Message-Passing (JSDMP), a new learning paradigm for rich-text graph representation learning. Besides considering similarity regarding structure and text, JSDMP further captures their corresponding dissimilarity by Jensen-Shannon divergence. Similarity and dissimilarity are then jointly used to compute new message weights among text nodes, thus enabling representations to learn with contextual and structural information from truly correlated text nodes. With JSDMP, we propose two novel graph neural networks, namely Divergent message-passing graph convolutional network (DMPGCN) and Divergent message-passing Page-Rank graph neural networks (DMPPRG), for learning representations in rich-text graphs. DMPGCN and DMPPRG have been extensively texted on well-established rich-text datasets and compared with several state-of-the-art baselines. The experimental results show that DMPGCN and DMPPRG can outperform other baselines, demonstrating the effectiveness of the proposed Jensen-Shannon Divergence Message-Passing paradigm

Jensen-Shannon Divergence Message-Passing for Rich-Text Graph Representation Learning

TL;DR

This work addresses representation learning in rich-text graphs by jointly modeling contextual and structural divergence alongside traditional similarity, via Jensen-Shannon Divergence Message-Passing (JSDMP). It introduces two Divergent Message-Passing GNNs, DMPGCN and DMPPRG, which compute divergence-aware edge weights to guide aggregation and feature fusion. Empirical results on six real-world datasets show that the proposed models outperform strong baselines, with notable gains in ACC and NMI across scientific articles, blogs, news, and Wikipedia graphs. The approach offers a principled way to leverage both content and structure in rich-text graphs and is extensible to larger language models for downstream NLP tasks.

Abstract

In this paper, we investigate how the widely existing contextual and structural divergence may influence the representation learning in rich-text graphs. To this end, we propose Jensen-Shannon Divergence Message-Passing (JSDMP), a new learning paradigm for rich-text graph representation learning. Besides considering similarity regarding structure and text, JSDMP further captures their corresponding dissimilarity by Jensen-Shannon divergence. Similarity and dissimilarity are then jointly used to compute new message weights among text nodes, thus enabling representations to learn with contextual and structural information from truly correlated text nodes. With JSDMP, we propose two novel graph neural networks, namely Divergent message-passing graph convolutional network (DMPGCN) and Divergent message-passing Page-Rank graph neural networks (DMPPRG), for learning representations in rich-text graphs. DMPGCN and DMPPRG have been extensively texted on well-established rich-text datasets and compared with several state-of-the-art baselines. The experimental results show that DMPGCN and DMPPRG can outperform other baselines, demonstrating the effectiveness of the proposed Jensen-Shannon Divergence Message-Passing paradigm
Paper Structure (21 sections, 10 equations, 4 figures, 3 tables)

This paper contains 21 sections, 10 equations, 4 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. Related works
  3. Rich-text representation learning
  4. Graph neural networks
  5. Jensen-Shannon Divergence message-passing graph neural networks for rich-text representation learning
  6. Notations
  7. Jensen-Shannon Divergence message-passing paradigm
  8. Divergent message-passing graph neural networks
  9. The learning objective
  10. Experiments and analysis
  11. Experimental setup
  12. Datasets
  13. Configurations of compared baselines
  14. Configurations of DMPGCN and DMPPRG
  15. Data spilt and evaluation metrics
  16. ...and 6 more sections

Figures (4)

  • Figure 1: The general architecture of existing GNN-based approaches to text representation learning with rich-text graph data. Here, we use news pages (nodes) and corresponding references/links (edges) to construct an exemplified rich-text graph as input.
  • Figure 2: Overall idea of the proposed Divergent message-passing graph convolutional network (DMPGCN). $\mathbf F$ and $\mathbf X$ represent node features matrix and latent position matrix, respectively.
  • Figure 3: Overall idea of the proposed Divergent message-passing graph Page-Rank network (DMPPRG).
  • Figure 4: Performance comparisons (ACC) of different variants of DMPGCN.