Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Asymptotic generalization error of a single-layer graph convolutional network

O. Duranthon, L. Zdeborová

TL;DR

This article predicts the performances of a single-layer graph convolutional network (GCN) trained on data produced by attributed stochastic block models (SBMs) in the high-dimensional limit and details the convergence rates of the GCN.

Abstract

While graph convolutional networks show great practical promises, the theoretical understanding of their generalization properties as a function of the number of samples is still in its infancy compared to the more broadly studied case of supervised fully connected neural networks. In this article, we predict the performances of a single-layer graph convolutional network (GCN) trained on data produced by attributed stochastic block models (SBMs) in the high-dimensional limit. Previously, only ridge regression on contextual-SBM (CSBM) has been considered in Shi et al. 2022; we generalize the analysis to arbitrary convex loss and regularization for the CSBM and add the analysis for another data model, the neural-prior SBM. We also study the high signal-to-noise ratio limit, detail the convergence rates of the GCN and show that, while consistent, it does not reach the Bayes-optimal rate for any of the considered cases.

Asymptotic generalization error of a single-layer graph convolutional network

TL;DR

This article predicts the performances of a single-layer graph convolutional network (GCN) trained on data produced by attributed stochastic block models (SBMs) in the high-dimensional limit and details the convergence rates of the GCN.

Abstract

While graph convolutional networks show great practical promises, the theoretical understanding of their generalization properties as a function of the number of samples is still in its infancy compared to the more broadly studied case of supervised fully connected neural networks. In this article, we predict the performances of a single-layer graph convolutional network (GCN) trained on data produced by attributed stochastic block models (SBMs) in the high-dimensional limit. Previously, only ridge regression on contextual-SBM (CSBM) has been considered in Shi et al. 2022; we generalize the analysis to arbitrary convex loss and regularization for the CSBM and add the analysis for another data model, the neural-prior SBM. We also study the high signal-to-noise ratio limit, detail the convergence rates of the GCN and show that, while consistent, it does not reach the Bayes-optimal rate for any of the considered cases.
Paper Structure (27 sections, 69 equations, 10 figures, 1 table)

This paper contains 27 sections, 69 equations, 10 figures, 1 table.

Table of Contents

  1. Introduction and related work
  2. Tight analysis in the high-dimensional limit:
  3. Generalization in graph neural networks:
  4. Main contribution:
  5. Models, setup
  6. Attributed SBMs:
  7. Analyzed GCN architecture:
  8. Bayes-optimal performances:
  9. Asymptotic prediction of the performances of the GCN
  10. Outline of the derivation:
  11. Consequences
  12. Conclusion
  13. Replica computation
  14. Gaussian equivalence
  15. CSBM
  16. ...and 12 more sections

Figures (10)

  • Figure 1: Search for the optimal parameters of the GCN on CSBM. $\alpha=4$, $\rho=0.1$. Top: low snr, $\lambda=0.5$, $\mu=1$. Bottom: high snr, $\lambda=1.5$, $\mu=3$. Full lines: prediction for the test accuracy obtained by eqs. \ref{['eq:formuleAccErr']} and \ref{['eq:répCsbmDébut']}-\ref{['eq:répCsbmFin']}; dots: numerical simulation of the GCN for $N=10^4$ and $d=30$, averaged over ten experiments; dotted line: Bayes-optimal test accuracy.
  • Figure 2: Search for the optimal parameters of the GCN on GLM--SBM. $\alpha=4$, $\rho=0.1$. Top: low snr, $\lambda=0.5$. Bottom: high snr, $\lambda=1.5$. Full lines: prediction for the test accuracy obtained by eqs. \ref{['eq:formuleAccErr']} and \ref{['eq:répGlmSbmDébut']}-\ref{['eq:répGlmSbmFin']}; dots: numerical simulation of the GCN for $N=10^4$ and $d=30$, averaged over ten experiments; dotted line: Bayes-optimal test accuracy.
  • Figure 3: Asymptotic misclassification error $1-\mathrm{Acc}_\mathrm{test}$; left: on the CSBM, $\alpha=4$; right: on the GLM--SBM. $r=10^3$, $\rho=0.1$. Dots: prediction for the test accuracy obtained by eqs. \ref{['eq:formuleAccErr']}, \ref{['eq:répCsbmDébut']}-\ref{['eq:répCsbmFin']} and \ref{['eq:répGlmSbmDébut']}-\ref{['eq:répGlmSbmFin']}, for $c=c^*$ optimal obtained by grid search. Dotted lines are given by \ref{['eq:tauxLgrand']} (for $\tau_\mathrm{CSBM}^\infty$) and \ref{['eq:tauxGlmSbm']} (for $\tau_\mathrm{GLM-SBM}$). The Bayes-optimal values are obtained from the equations given in appendix \ref{['sec:eqBO']}.
  • Figure 4: Optimal self-loop strength $c^*$ vs graph snr $\lambda$. $d=30$, $\rho=0.1$, $r=10^3$ and $l$ quadratic. Left: on the CSBM, $N=10^4$, $\alpha=4$, $\mu=3$. Right: on the fashion-SBM, classes 2 and 4. The lines are numerical simulations of the GCN averaged over ten experiments. $c^*$ is computed as the extremizer of the simulated $\mathrm{Acc}_\mathrm{test}$.
  • Figure 5: Search for the optimal parameters of the GCN. $\alpha=0.7$, $\rho=0.1$. Top: CSBM, $\lambda=1.5$, $\mu=3$. Bottom: GLM--SBM, $\lambda=1$. Full lines: prediction for the test accuracy obtained by eqs. \ref{['eq:formuleAccErr']}; dots: numerical simulation of the GCN for $N=10^4$ and $d=30$, averaged over ten experiments; dotted line: Bayes-optimal test accuracy.
  • ...and 5 more figures