Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Graph Attention Network for Node Regression on Random Geometric Graphs with Erdős--Rényi contamination

Somak Laha, Suqi Liu, Morgane Austern

TL;DR

The paper addresses node regression on latent-position graphs perturbed by ER contamination by introducing a discretized attention scheme that builds denoised proxies for latent covariates. By crossing screening and averaging across two blocks of features, the proposed proxies $\boldsymbol{\lambda}_i$ converge to the true covariates $\mathbf{x}_i$, enabling consistent estimation of $\boldsymbol{\beta}$ via proxy regression and improved prediction for unseen nodes. The work provides rigorous asymptotic guarantees, showing an explicit regime where the proxy-based estimator outperforms non-attention GNNs and yields an oracle-like MSE for unlabelled-node prediction, corroborated by extensive synthetic and real-data experiments. Overall, the results establish a provable advantage of discretized attention over vanilla averaging in noisy, contaminated graphs and suggest a robust preprocessing pathway for other downstream graph tasks.

Abstract

Graph attention networks (GATs) are widely used and often appear robust to noise in node covariates and edges, yet rigorous statistical guarantees demonstrating a provable advantage of GATs over non-attention graph neural networks~(GNNs) are scarce. We partially address this gap for node regression with graph-based errors-in-variables models under simultaneous covariate and edge corruption: responses are generated from latent node-level covariates, but only noise-perturbed versions of the latent covariates are observed; and the sample graph is a random geometric graph created from the node covariates but contaminated by independent Erdős--Rényi edges. We propose and analyze a carefully designed, task-specific GAT that constructs denoised proxy features for regression. We prove that regressing the response variables on the proxies achieves lower error asymptotically in (a) estimating the regression coefficient compared to the ordinary least squares (OLS) estimator on the noisy node covariates, and (b) predicting the response for an unlabelled node compared to a vanilla graph convolutional network~(GCN) -- under mild growth conditions. Our analysis leverages high-dimensional geometric tail bounds and concentration for neighbourhood counts and sample covariances. We verify our theoretical findings through experiments on synthetically generated data. We also perform experiments on real-world graphs and demonstrate the effectiveness of the attention mechanism in several node regression tasks.

Graph Attention Network for Node Regression on Random Geometric Graphs with Erdős--Rényi contamination

TL;DR

The paper addresses node regression on latent-position graphs perturbed by ER contamination by introducing a discretized attention scheme that builds denoised proxies for latent covariates. By crossing screening and averaging across two blocks of features, the proposed proxies converge to the true covariates , enabling consistent estimation of via proxy regression and improved prediction for unseen nodes. The work provides rigorous asymptotic guarantees, showing an explicit regime where the proxy-based estimator outperforms non-attention GNNs and yields an oracle-like MSE for unlabelled-node prediction, corroborated by extensive synthetic and real-data experiments. Overall, the results establish a provable advantage of discretized attention over vanilla averaging in noisy, contaminated graphs and suggest a robust preprocessing pathway for other downstream graph tasks.

Abstract

Graph attention networks (GATs) are widely used and often appear robust to noise in node covariates and edges, yet rigorous statistical guarantees demonstrating a provable advantage of GATs over non-attention graph neural networks~(GNNs) are scarce. We partially address this gap for node regression with graph-based errors-in-variables models under simultaneous covariate and edge corruption: responses are generated from latent node-level covariates, but only noise-perturbed versions of the latent covariates are observed; and the sample graph is a random geometric graph created from the node covariates but contaminated by independent Erdős--Rényi edges. We propose and analyze a carefully designed, task-specific GAT that constructs denoised proxy features for regression. We prove that regressing the response variables on the proxies achieves lower error asymptotically in (a) estimating the regression coefficient compared to the ordinary least squares (OLS) estimator on the noisy node covariates, and (b) predicting the response for an unlabelled node compared to a vanilla graph convolutional network~(GCN) -- under mild growth conditions. Our analysis leverages high-dimensional geometric tail bounds and concentration for neighbourhood counts and sample covariances. We verify our theoretical findings through experiments on synthetically generated data. We also perform experiments on real-world graphs and demonstrate the effectiveness of the attention mechanism in several node regression tasks.
Paper Structure (41 sections, 55 theorems, 400 equations, 2 figures, 2 tables, 2 algorithms)

This paper contains 41 sections, 55 theorems, 400 equations, 2 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Notations.
  3. Related work
  4. Node regression on random graphs
  5. Graph distribution
  6. Covariates and response distribution
  7. Asymptotic regime and parameters
  8. Problem statement.
  9. Main results
  10. Estimation algorithm through discretized attention-based proxies
  11. Efficient estimation guarantee of
  12. Prediction algorithm for unlabelled node
  13. Prediction MSE guarantee
  14. Prediction lower bound for graph convolutional network
  15. Auxiliary result on denoising guarantee
  16. ...and 26 more sections

Key Result

Proposition 4.1

Assume that $d\ll n$. Then the OLS estimator $\hat{\bm \beta}_z = (\bm Z_n^\top \bm Z_n)^{-1}\bm Z_n^\top \bm Y_n$ is not consistent for the parameter $\bm \beta$, in the sense that, as $n \longrightarrow \infty$,

Figures (2)

  • Figure 1: Comparison of relative $L_2$ errors of the estimates $\hat{\bm \beta}_\lambda$ and $\hat{\bm \beta}_z$ by varying $\gamma$ and $\sigma_\eta^2$ respectively. We fix the parameters $n=30000, d=250, \sigma_x^2=1,\sigma_\varepsilon^2=1,$ and $\alpha=0.72$. In the first plot, we vary $\gamma$ from $0.70$ to $0.75$ in $0.05$ increments and fix $\sigma_\eta^2=1$. In the second plot, we vary $\sigma_\eta^2$ from $0.25$ to $3.00$ in $0.25$ increments and fix $\gamma=0.725$. Curves show seed-averaged relative errors (mean $\pm \mathrm{SD}$ ) across $10$ seeds.
  • Figure 2: MSE comparison for graph-based regression prediction on OGBN-Products, OGBN-MAG (paper), and PyG-Reddit under ER edge contamination and covariate noise ($\sigma_\eta^2$). Methods: OLS on noisy covariates, GCN, GAT, and proxy-regression (ours). Curves show seed-averaged MSE (mean $\pm \,\mathrm{SD}$) across $10$ seeds. Top row evaluates all nodes; bottom row evaluates high-degree nodes only.

Theorems & Definitions (104)

  • Definition 3.1: Erdős--Rényi contaminated random dot-product graph
  • Proposition 4.1
  • Theorem 4.2
  • Theorem 4.3
  • Theorem 4.4
  • Lemma 4.5
  • Lemma 2.1
  • proof
  • Theorem 2.2
  • proof
  • ...and 94 more