A Simple Spectral Failure Mode for Graph Convolutional Networks

TL;DR

This short paper presents a simple generative model where unsupervised graph convolutional network fails, while the adjacency spectral embedding succeeds, and demonstrates the phenomenon of missing inference signals in non-leading eigenvectors.

Abstract

Neural networks have achieved remarkable successes in machine learning tasks. This has recently been extended to graph learning using neural networks. However, there is limited theoretical work in understanding how and when they perform well, especially relative to established statistical learning techniques such as spectral embedding. In this short paper, we present a simple generative model where unsupervised graph convolutional network fails, while the adjacency spectral embedding succeeds. Specifically, unsupervised graph convolutional network is unable to look beyond the first eigenvector in certain approximately regular graphs, thus missing inference signals in non-leading eigenvectors. The phenomenon is demonstrated by visual illustrations and comprehensive simulations.

Figures (4)

• Figure 1: Geometry for the canonical case where ASE succeeds but GCN fails. The top panel shows the density: the black line represents density of $f_X$, while the green line and red line represent the class-conditional density of $f_{X|0}$ and $f_{X|1}$. The bottom panel shows one sample data generation with $n=1000$ and $m=100$: the green crosses and red crosses are training points from $f_{X|0}$ and $f_{X|1}$ respectively with known label, while the black crosses are the remaining test points. Round dots are ASE after Procrustes with same color setting. Contours are from kernel density estimation of red dots and green dots. For this case, indicated by the vertical line at $|\theta_E - \theta_\perp| = \pi/32$ in Figure \ref{['fig3']}, both ASE into dimension $d'=1$ and unsupervised two-layer GCN perform poorly while ASE into two dimensions has nearly optimal performance.
• Figure 2: Performance for the canonical case where ASE succeeds but two-layer unsupervised GCN fails. ERM in the legend stands for empirical risk minimization classifier. We run $100$ Monte Carlo replicates and report the average classification error.
• Figure 3: Effects of changing $m/n$ on the classification performance for different methods. Vertical dash line indicates $m/n = 0.1$ used in Figure \ref{['fig3']}.
• Figure 4: Adding parameter cross validation results for unsupervised GCN, and include semisupervised GCN. The curves for GCN models are obtained without hyper-parameter search, while the stars represent the best results after hyper-parameter search.