Scalable Weibull Graph Attention Autoencoder for Modeling Document Networks

TL;DR

The paper addresses the limitations of Gaussian-parameterized VGAEs in modeling document relational networks by introducing a non-Gaussian, deep RTM-based decoder (GPFA/GPGBN) and pairing it with Weibull-based encoders to form Weibull Graph Autoencoders (WGAEs). The GPFA provides analytic posteriors for joint modeling of node features and links, while GPGBN extends this to a multi-layer hierarchical RTM that captures hierarchical semantic topics and multilevel relationships. Two encoder variants—a vanilla graph convolutional encoder (WGCAE) and a Bayesian attention-based encoder (WGAAE)—are integrated with the GPGBN decoder, and training supports both full-batch and scalable mini-batch regimes, including a subgraph decoding strategy to reduce cost. Experimental results demonstrate stronger hierarchical latent representations and competitive performance on link prediction, clustering, and classification tasks, with scalability to large graphs such as MAG240M. Overall, the work offers a scalable, interpretable framework for multilevel DRN analysis that combines deep RTMs with flexible, non-Gaussian variational inference.

Abstract

Although existing variational graph autoencoders (VGAEs) have been widely used for modeling and generating graph-structured data, most of them are still not flexible enough to approximate the sparse and skewed latent node representations, especially those of document relational networks (DRNs) with discrete observations. To analyze a collection of interconnected documents, a typical branch of Bayesian models, specifically relational topic models (RTMs), has proven their efficacy in describing both link structures and document contents of DRNs, which motives us to incorporate RTMs with existing VGAEs to alleviate their potential issues when modeling the generation of DRNs. In this paper, moving beyond the sophisticated approximate assumptions of traditional RTMs, we develop a graph Poisson factor analysis (GPFA), which provides analytic conditional posteriors to improve the inference accuracy, and extend GPFA to a multi-stochastic-layer version named graph Poisson gamma belief network (GPGBN) to capture the hierarchical document relationships at multiple semantic levels. Then, taking GPGBN as the decoder, we combine it with various Weibull-based graph inference networks, resulting in two variants of Weibull graph auto-encoder (WGAE), equipped with model inference algorithms. Experimental results demonstrate that our models can extract high-quality hierarchical latent document representations and achieve promising performance on various graph analytic tasks.

Figures (7)

• Figure 1: The probabilistic generative process of (a) Graph Poisson Graph Factor Analysis (GPFA) and (b) Graph Poisson Gamma Belief Network (GPGBN).
• Figure 2: Illustration of the network structure of a 3-layer Weibull graph autoencoder (WGAE), which consists of a Weibull-based graph inference network (encoder) on the left and a GPGBN (decoder) on the right.
• Figure 3: Visualization of hierarchical semantic topics learned by 3-layer GPGBNs on 20News dataset.
• Figure 4: Visualization of document subnetworks learned by a 3-layer GPGBN on 20News dataset. For each subnetwork, taking the $i$th document as the source node, other documents, whose connections at layer $t$ (denoted as ${u_k^{(t)}\theta _{ik}^{(t)}\theta _{jk}^{(t)}}$) are larger than a threshold $\tau_u$, are displayed in the black boxes, and the key words of connected topics are displayed in the blue, green, and red text boxes, respectively, from shallow to deep. The common key words simultaneously appear in both documents connected with their associated topics are highlighted with the corresponding topic color.
• Figure 5: Visualization of part (randomly selected 25 nodes) of the hierarchical relationships learned by 3-layer WGCAEs on Cora (the first row) and Citeseer (the second row). The first column represents the observed adjacency matrix ${\bf A}$ and the second to fourth columns represent the learned adjacency matrices ${\bf A}^{(t)}$ from the layer 1 to 3, respectively. After normalization, a brighter point of each ${\bf A}^{(t)}$ indicates a stronger node relationship, and the red zone is highlighted for better demonstrations.