HierPromptLM: A Pure PLM-based Framework for Representation Learning on Heterogeneous Text-rich Networks

TL;DR

This work tackles representation learning on heterogeneous text-rich networks by proposing HierPromptLM, a pure PLM-based framework that unifies textual and heterogeneous structural information in a single textual space. It introduces a Hierarchical Prompt module, including a graph-aware prompt built from meta-path–based subgraphs and a relation-aware prompt with a learnable relation token, to jointly encode node- and edge-level heterogeneity. Two specialized pretraining tasks, HGA-NSP and HGA-MLM, are designed to reinforce the interactions between text and structure, leading to strong gains on node classification and link prediction across DBLP and OAG datasets. The method demonstrates robustness through training-free variants and backbone extensions (e.g., T5-base, GPT-2-small), highlighting the practical impact of a unified PLM-based approach for heterogeneous, text-rich graphs.

Abstract

Representation learning on heterogeneous text-rich networks (HTRNs), which consist of multiple types of nodes and edges with each node associated with textual information, is essential for various real-world applications. Given the success of pretrained language models (PLMs) in processing text data, recent efforts have focused on integrating PLMs into HTRN representation learning. These methods typically handle textual and structural information separately, using both PLMs and heterogeneous graph neural networks (HGNNs). However, this separation fails to capture the critical interactions between these two types of information within HTRNs. Additionally, it necessitates an extra alignment step, which is challenging due to the fundamental differences between distinct embedding spaces generated by PLMs and HGNNs. To deal with it, we propose HierPromptLM, a novel pure PLM-based framework that seamlessly models both text data and graph structures without the need for separate processing. Firstly, we develop a Hierarchical Prompt module that employs prompt learning to integrate text data and heterogeneous graph structures at both the node and edge levels, within a unified textual space. Building upon this foundation, we further introduce two innovative HTRN-tailored pretraining tasks to fine-tune PLMs for representation learning by emphasizing the inherent heterogeneity and interactions between textual and structural information within HTRNs. Extensive experiments on two real-world HTRN datasets demonstrate HierPromptLM outperforms state-of-the-art methods, achieving significant improvements of up to 6.08% for node classification and 10.84% for link prediction.

Figures (7)

• Figure 1: An illustrative example of an HTRN.
• Figure 2: The framework of HierPromptLM. (a) Meta-path-based graph token generation: extract meta-path-based subgraphs from the HTRN, create meaningful textual summaries of these subgraphs, and distill graph tokens using a frozen PLM. (b) Graph-aware prompt: for each target node, generate a text sequence by integrating its own text data with meta-path-based graph tokens and their corresponding descriptions. (c) Relation-aware prompt: for each target edge, generate a text sequence by combining each node's graph-aware prompt with a learnable relation token that connects them. (d) HTRN-tailored pretraining: fine-tune a PLM with the relation-aware prompt using two specialized pretraining tasks. (e) Downstream tasks.
• Figure 3: (a) Training-free extension on OAG. (b) PLM backbone extension on DBLP. (c) PLM backbone extension on OAG.
• Figure 4: Training-free extension on DBLP.
• Figure 5: Impact of pre-defined meta-path.