Learning Multiplex Representations on Text-Attributed Graphs with One Language Model Encoder

TL;DR

METAG tackles multiplex text-attributed graphs by learning relation-conditioned embeddings with a single shared language model encoder augmented by relation-prior tokens. It enables direct inference for singular relations and learn-to-select-source-relations when relations are mixed, using a relation-aware objective and efficient negative sampling. Across five graphs and nine tasks, METAG consistently outperforms strong baselines, achieving superior multiplex representations while maintaining practical time and memory costs. The approach provides a scalable, interpretable framework to exploit diverse semantic relations in text-rich graphs, with strong implications for academic and e-commerce applications.

Abstract

In real-world scenarios, texts in a graph are often linked by multiple semantic relations (e.g., papers in an academic graph are referenced by other publications, written by the same author, or published in the same venue), where text documents and their relations form a multiplex text-attributed graph. Mainstream text representation learning methods use pretrained language models (PLMs) to generate one embedding for each text unit, expecting that all types of relations between texts can be captured by these single-view embeddings. However, this presumption does not hold particularly in multiplex text-attributed graphs. Along another line of work, multiplex graph neural networks (GNNs) directly initialize node attributes as a feature vector for node representation learning, but they cannot fully capture the semantics of the nodes' associated texts. To bridge these gaps, we propose METAG, a new framework for learning Multiplex rEpresentations on Text-Attributed Graphs. In contrast to existing methods, METAG uses one text encoder to model the shared knowledge across relations and leverages a small number of parameters per relation to derive relation-specific representations. This allows the encoder to effectively capture the multiplex structures in the graph while also preserving parameter efficiency. We conduct experiments on nine downstream tasks in five graphs from both academic and e-commerce domains, where METAG outperforms baselines significantly and consistently. The code is available at https://github.com/PeterGriffinJin/METAG.

Figures (10)

• Figure 1: (a) Raw data distribution shift and (b) performance shift across different relations on a graph of Geology papers. cb, sa, sv, cr, and ccb represent cited-by, same-author, same-venue, co-reference, and co-cited-by relation, respectively. Each entry in (a) and (b) are the Jaccard score and the PREC@1 of BERT embedding fine-tuned on the corresponding source relation distribution and tested on the corresponding target relation distribution.
• Figure 2: Model Framework Overview. Metag has a language model encoder to model the shared knowledge among relations and relation-prior embeddings to capture the relation-specific signals.
• Figure 3: The learned source relation weights for citation prediction and paper recommendation on the Geology graph.
• Figure 4: Multiplex embedding visualization on the Geology graph. cb, sa, and sv represent the "cited-by", "same-author", and "same-venue" relation respectively.
• Figure 5: Analysis of relation weights $w_r$ on Geology. The x-axis is the training step and the y-axis is MRR on the validation set. We compare uniform $w_r$ with a better-tuned $w_r$.

Theorems & Definitions (2)

• Definition 2.1
• Definition 2.2