HyperBERT: Mixing Hypergraph-Aware Layers with Language Models for Node Classification on Text-Attributed Hypergraphs

TL;DR

A new architecture, HyperBERT, a mixed text-hypergraph model which simultaneously models hypergraph relational structure while maintaining the high-quality text encoding capabilities of a pre-trained BERT is proposed.

Abstract

Hypergraphs are characterized by complex topological structure, representing higher-order interactions among multiple entities through hyperedges. Lately, hypergraph-based deep learning methods to learn informative data representations for the problem of node classification on text-attributed hypergraphs have garnered increasing research attention. However, existing methods struggle to simultaneously capture the full extent of hypergraph structural information and the rich linguistic attributes inherent in the nodes attributes, which largely hampers their effectiveness and generalizability. To overcome these challenges, we explore ways to further augment a pretrained BERT model with specialized hypergraph-aware layers for the task of node classification. Such layers introduce higher-order structural inductive bias into the language model, thus improving the model's capacity to harness both higher-order context information from the hypergraph structure and semantic information present in text. In this paper, we propose a new architecture, HyperBERT, a mixed text-hypergraph model which simultaneously models hypergraph relational structure while maintaining the high-quality text encoding capabilities of a pre-trained BERT. Notably, HyperBERT presents results that achieve a new state-of-the-art on five challenging text-attributed hypergraph node classification benchmarks.

Figures (2)

• Figure 1: An illustration of a standard text-attributed graph (left) where nodes are connected between each other using binary relations, and a text-attributed hypergraph (right) where nodes are related with high-order connections (hyperedges). In both cases, each node is attributed with a textual description, such as paper abstract in the case for co-citation hypergraphs.
• Figure 2: High-level overview of our proposed HyperBERT model. The model mixes hypergraph-aware layers into BERT to simultaneously exploit hypergraph topology and text semantics for node classification tasks on TAHGs. To accomplish the text-hypergraph alignment in the feature space, HyperBERT employs a novel self-supervised loss for pretraining that effectively aligns semantic and hypergraph feature spaces. After pretraining the model, it can be fine-tuned on a variety of downstream hypergraph node-level tasks such as node classification.