Relieving the Over-Aggregating Effect in Graph Transformers

TL;DR

The paper identifies over-aggregating as a key failure mode in graph transformers with global attention, where many messages are pooled with little discrimination as the graph size grows. It proposes Wideformer, a plug-and-play approach that divides aggregation into multiple clusters and guides attention to the most informative clusters, preserving a global receptive field while reducing input dilution. The method is evaluated across thirteen real-world datasets with three linear backbones, showing consistent reductions in attention entropy and substantive performance gains over strong baselines; complexity scales as $O(nm)$ with modest overhead. The work provides extensive ablations and analyses on informative source nodes, cluster sizing, and center selection, and releases code to facilitate reproducibility and broader adoption. Overall, Wideformer offers a practical path to mitigating over-aggregating in graph transformers and enhancing their applicability to large-scale graphs.

Abstract

Graph attention has demonstrated superior performance in graph learning tasks. However, learning from global interactions can be challenging due to the large number of nodes. In this paper, we discover a new phenomenon termed over-aggregating. Over-aggregating arises when a large volume of messages is aggregated into a single node with less discrimination, leading to the dilution of the key messages and potential information loss. To address this, we propose Wideformer, a plug-and-play method for graph attention. Wideformer divides the aggregation of all nodes into parallel processes and guides the model to focus on specific subsets of these processes. The division can limit the input volume per aggregation, avoiding message dilution and reducing information loss. The guiding step sorts and weights the aggregation outputs, prioritizing the informative messages. Evaluations show that Wideformer can effectively mitigate over-aggregating. As a result, the backbone methods can focus on the informative messages, achieving superior performance compared to baseline methods.

Figures (12)

• Figure 1: Over-Aggregating in Graph Transformers. The entropy values are normalized to $[0,1]$, with higher entropy indicating a more uniform distribution of attention scores. In (b), we show the attention entropy with different numbers of nodes for Polynormer deng_PolynormerPolynomialExpressiveGraph_2023, a typical linear attention method.
• Figure 2: Effect of Over-Aggregating by Attention Entropy Regularization (Measured by accuracy except for ROC-AUC for minesweeper: %).
• Figure 3: Attention Comparison between Global Aggregation and Our Divided Global Aggregation. Global attention employs global aggregation to aggregate source messages into a one-dimensional representation. To relieve the over-aggregating, we assign source nodes into distinct clusters and aggregate each cluster separately, giving rise to multiple aggregation results.
• Figure 4: Effectiveness Evaluation for Over-Aggregating. "+ Ours" denotes using our proposed Wideformer. "+ Ours w/o. Guide" employs Wideformer without attention guidance.
• Figure 5: Informative Source Node Ratio. The ratio is the number of nodes assigned to the cluster with the highest attention score relative to the total number of graph nodes.

Theorems & Definitions (4)

• Theorem 3.1: Monotonic Lower Bond of Attention Entropy
• Lemma A.1: Lower Bound of Attention Entropy
• proof
• proof