In-n-Out: Calibrating Graph Neural Networks for Link Prediction

TL;DR

IN-N-OUT is the first-ever method to calibrate GNNs for link prediction, based on two simple intuitions: attributing true/false labels to an edge while respecting a GNNs prediction should cause but small fluctuations in that edge's embedding, and if that same edge contradicting the authors' GNN, embeddings should change more substantially.

Abstract

Deep neural networks are notoriously miscalibrated, i.e., their outputs do not reflect the true probability of the event we aim to predict. While networks for tabular or image data are usually overconfident, recent works have shown that graph neural networks (GNNs) show the opposite behavior for node-level classification. But what happens when we are predicting links? We show that, in this case, GNNs often exhibit a mixed behavior. More specifically, they may be overconfident in negative predictions while being underconfident in positive ones. Based on this observation, we propose IN-N-OUT, the first-ever method to calibrate GNNs for link prediction. IN-N-OUT is based on two simple intuitions: i) attributing true/false labels to an edge while respecting a GNNs prediction should cause but small fluctuations in that edge's embedding; and, conversely, ii) if we label that same edge contradicting our GNN, embeddings should change more substantially. An extensive experimental campaign shows that IN-N-OUT significantly improves the calibration of GNNs in link prediction, consistently outperforming the baselines available -- which are not designed for this specific task.

Figures (4)

• Figure 1: Reliability diagrams of VGAE, PEG and SAGE trained on the cora, citeseer, and pubmed datasets. The dashed line is a baseline for perfect calibration. In most cases, the GNNs tend to attribute higher probability than they should to the positive class. On one hand, this hints at over-confident predictions for edges positively classified. On the other, this indicates under-confidence in our negative predictions. This is behavior is notably more complex than what was previously observed for GNN-based node classification CaGCNGATS, in which predictions are overall underconfident.
• Figure 2: Post-calibration reliability diagrams using IN-N-OUT and off-the-shelf calibration baselines. IN-N-OUT decreases the ECE of the original GNN models and approaches better the identity lines (denoting a perfectly calibrated model).
• Figure 3: Pre-calibration reliability diagram for all models and datasets.
• Figure 4: Post calibration reliability diagram for GCN and all datasets. Overall, IN-N-OUT reduced the ECE of all GNNs, resulting in more well-behaved reliability diagrams when compared to \ref{['fig:pre-calibration-all']}.