Wide & Deep Learning for Node Classification

TL;DR

This work addresses the challenge of learning robust node representations by balancing memorization and generalization in graph neural networks. It introduces GCNIII, a Wide & Deep architecture that combines a linear wide component with a deep GCN-based component, and embeds three techniques—Intersect memory, Initial residual, and Identity mapping—as hyperparameters to control information flow and generalization. The paper demonstrates state-of-the-art results across semi- and full-supervised node classification tasks and confirms the value of node features, including LLM-generated features, through extensive ablative and inductive-learning experiments. The findings highlight that effective dropout and attaching information-rich features near the output are key to preventing over-generalization in deep GCNs, with practical impact for cross-domain graph classification and real-world graph analysis.

Abstract

Wide & Deep, a simple yet effective learning architecture for recommendation systems developed by Google, has had a significant impact in both academia and industry due to its combination of the memorization ability of generalized linear models and the generalization ability of deep models. Graph convolutional networks (GCNs) remain dominant in node classification tasks; however, recent studies have highlighted issues such as heterophily and expressiveness, which focus on graph structure while seemingly neglecting the potential role of node features. In this paper, we propose a flexible framework GCNIII, which leverages the Wide & Deep architecture and incorporates three techniques: Intersect memory, Initial residual and Identity mapping. We provide comprehensive empirical evidence showing that GCNIII can more effectively balance the trade-off between over-fitting and over-generalization on various semi- and full- supervised tasks. Additionally, we explore the use of large language models (LLMs) for node feature engineering to enhance the performance of GCNIII in cross-domain node classification tasks. Our implementation is available at https://github.com/CYCUCAS/GCNIII.

Figures (6)

• Figure 1: Training error and validation error of the semi-supervised task on Cora with 2-layer vanilla GCN (left) and 64-layer GCNII (right). The training error of deep GCNII is very volatile and much higher than the validation error. We call this phenomenon over-generalization.
• Figure 2: Wide & Deep architecture GCNIII.
• Figure 3: LLM for GCNIII. Sparse discrete features of graph nodes can be constructed using LLM, such as bag-of-words representation of document, which can be used in both the Wide and Deep Components. A unified text-attribute description format can also be used to construct text-attribute graphs (TAGs) as input to the LLM, generating dense continuous features that enhance the learning of the Deep Component.
• Figure 4: Example network architecture for 64-layer GCNII. The color gradient from black to white represents the weight $\beta_l$ of the Linear Transformation from large to small. Initial residual inputs $\mathbf{H}^{(0)}$ directly to each layer, and the network between layers 56 and 64 contains an 8-layer sub-GCNII. This structure is similar to a reversed JKNet.
• Figure 5: Training error of the semi-supervised task on Cora with GCN, GCNII and GCNIII.

Theorems & Definitions (3)

• Theorem 4.1
• Theorem 1.1
• proof