Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Graph Neural Networks can Recover the Hidden Features Solely from the Graph Structure

Ryoma Sato

TL;DR

These results show that GNNs can fully exploit the graph structure by themselves, and in effect, Gnns can use both the hidden and explicit node features for downstream tasks.

Abstract

Graph Neural Networks (GNNs) are popular models for graph learning problems. GNNs show strong empirical performance in many practical tasks. However, the theoretical properties have not been completely elucidated. In this paper, we investigate whether GNNs can exploit the graph structure from the perspective of the expressive power of GNNs. In our analysis, we consider graph generation processes that are controlled by hidden (or latent) node features, which contain all information about the graph structure. A typical example of this framework is kNN graphs constructed from the hidden features. In our main results, we show that GNNs can recover the hidden node features from the input graph alone, even when all node features, including the hidden features themselves and any indirect hints, are unavailable. GNNs can further use the recovered node features for downstream tasks. These results show that GNNs can fully exploit the graph structure by themselves, and in effect, GNNs can use both the hidden and explicit node features for downstream tasks. In the experiments, we confirm the validity of our results by showing that GNNs can accurately recover the hidden features using a GNN architecture built based on our theoretical analysis.

Graph Neural Networks can Recover the Hidden Features Solely from the Graph Structure

TL;DR

These results show that GNNs can fully exploit the graph structure by themselves, and in effect, Gnns can use both the hidden and explicit node features for downstream tasks.

Abstract

Graph Neural Networks (GNNs) are popular models for graph learning problems. GNNs show strong empirical performance in many practical tasks. However, the theoretical properties have not been completely elucidated. In this paper, we investigate whether GNNs can exploit the graph structure from the perspective of the expressive power of GNNs. In our analysis, we consider graph generation processes that are controlled by hidden (or latent) node features, which contain all information about the graph structure. A typical example of this framework is kNN graphs constructed from the hidden features. In our main results, we show that GNNs can recover the hidden node features from the input graph alone, even when all node features, including the hidden features themselves and any indirect hints, are unavailable. GNNs can further use the recovered node features for downstream tasks. These results show that GNNs can fully exploit the graph structure by themselves, and in effect, GNNs can use both the hidden and explicit node features for downstream tasks. In the experiments, we confirm the validity of our results by showing that GNNs can accurately recover the hidden features using a GNN architecture built based on our theoretical analysis.
Paper Structure (18 sections, 5 theorems, 41 equations, 4 figures, 1 table)

This paper contains 18 sections, 5 theorems, 41 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Graph Neural Networks and Its Theory
  4. Feature Recovery
  5. Background and Problem Formulation
  6. Assumptions
  7. Graph Neural Networks
  8. Why Is Recovery Challenging?
  9. Main Results
  10. Graph Neural Networks can Recover the Threshold Function
  11. Graph Neural Networks can Recover the Hidden Features
  12. Experiments
  13. Recovering Features
  14. Performance on Downstream Tasks
  15. Conclusion
  16. ...and 3 more sections

Key Result

Theorem 4.1

For any $s$ and $g$ that satisfy Assumptions 1-5, there exist $\theta_1, \theta_2, \ldots$ such that with the explicit node features ${\boldsymbol{X}}$ defined by Eq. eq: orig_feature, where the probability is with respect to the draw of samples ${\boldsymbol{z}}_1, {\boldsymbol{z}}_2, \ldots$.

Figures (4)

  • Figure 1: (a) Traditional View (Rich Node Features). GNNs filter features by mixing them with neighboring nodes. (b) Traditional View (Uninformative Features). Filters cannot generate informative features if the inputs are not informative, i.e., garbage in, garbage out. (c) Our Results. GNNs create informative node features by themselves even when the input node features are uninformative by absorbing information from the underlying graph. (d) Illustrations of the Problem Setting. (d.1) Nodes have hidden features from which the input graph is generated. (d.2) The input to GNNs is a vanilla graph without any additional features. Nodes have coordinates for visualization in this panel, but these coordinates are neither fed to GNNs. (d.3) GNNs try to recover the hidden features.
  • Figure 2: Illustrations of the Difficulty of Recovery. The input graph is $10$-NN graph of the hidden features. The shortest path distance between points A and B is 21 hops, and the shortest path distance between points A and C is 18 hops. These distances indicate that point C is closer to point A than point B, but this is not the case in the true feature space. Standard node embedding methods would embed node C closer to A than node B to A, which is not consistent with the true feature. Embedding nodes that are close in the input graph close is the critical assumption in various embedding methods. This assumption does NOT hold in our situation. This disagreement is caused by the different scales of edges in sparse and dense regions. The difficulty lies in the fact that these scales are not directly available in the input information.
  • Figure 3: Results for the Transductive Setting. Overall, the proposed method succeeded in recovering the ground truth hidden features while tSNE to ${\boldsymbol{X}}$ (Eq. \ref{['eq: extended_feature']}) fails, and GINs and GATs are mediocre. (Top Left) The ground truth hidden embeddings. The node ids are numbered based on the x-coordinate and shown in the node colors. These node ids are for visualization purposes only and are NOT shown to GNNs and downstream algorithms. (Top Mid) The input graph constructed from the hidden features. The positions of the visualization are NOT shown to GNNs. (Top Right) tSNE plot on the synthetic node features, i.e., Eq. \ref{['eq: extended_feature']}. These results indicate that the node features are not informative for feature recovery. This introduces challenges to the task. (Bottom Left) The recovered features by the proposed method. They resemble the ground truth not only with respect to the cluster structure but also the x-coordinates (shown in the node colors), the curved moon shapes in the two-moon dataset, and the striped pattern in the Adult dataset. The $d_G$ value (Eq. \ref{['eq: d_G']}) is small, which indicates the success of the recovery and validates the theory. (Bottom Mid) The recovered features by GINs. They do not resemble the true hidden features. The $d_G$ value is mediocre. (Bottom Right) The recovered features by GATs. They do not resemble hidden features, but some clusters are detected (shown in the node colors). The $d_G$ value is mediocre. These results show that existing GNNs can extract some information from the graph structure, but they do not fully recover the hidden features.
  • Figure 4: Results for the Inductive Setting. The legends and tendencies are the same as in Figure \ref{['fig: transductive']}. The proposed method succeeded in generalizing to different sizes and keeping $d_G$ low even in the extrapolation setting. GINs and GAT partially succeeded in extracting some of graph information, but they are not perfect, and $d_G$ is moderately high.

Theorems & Definitions (6)

  • Theorem 4.1
  • Theorem 4.2
  • Definition 4.3
  • Theorem 4.4
  • Theorem 4.5
  • Lemma 1.1