Cardinality-Preserving Structured Sparse Graph Transformers for Molecular Property Prediction

TL;DR

Cardinality-Preserving Structured Sparse Graph Transformers for Molecular Property Prediction tackles data scarcity in drug discovery by combining structured sparse attention with Graphormer biases and a novel cardinality-preserving attention channel. The method uses SPD-based locality (K-hop with $K=3$), direct-bond edge bias, and a CPA term to preserve dynamic neighborhood signals, trained with dual self-supervised objectives (contrastive graph-level alignment and masked attribute reconstruction) on ~28M molecules. Evaluation under a fully matched protocol across 11 benchmarks (MoleculeNet, OGB, TDC) shows statistically significant gains on most tasks over strong reproduced baselines. The work demonstrates robust data-efficient molecular representations and provides code and dataset details for reproducibility. This approach broadens the applicability of graph transformers in sparse data regimes for molecular property prediction.

Abstract

Drug discovery motivates efficient molecular property prediction under limited labeled data. Chemical space is vast, often estimated at approximately 10^60 drug-like molecules, while only thousands of drugs have been approved. As a result, self-supervised pretraining on large unlabeled molecular corpora has become essential for data-efficient molecular representation learning. We introduce **CardinalGraphFormer**, a graph transformer that incorporates Graphormer-inspired structural biases, including shortest-path distance and centrality, as well as direct-bond edge bias, within a structured sparse attention regime limited to shortest-path distance <= 3. The model further augments this design with a cardinality-preserving unnormalized aggregation channel over the same support set. Pretraining combines contrastive graph-level alignment with masked attribute reconstruction. Under a fully matched evaluation protocol, CardinalGraphFormer improves mean performance across all 11 evaluated tasks and achieves statistically significant gains on 10 of 11 public benchmarks spanning MoleculeNet, OGB, and TDC ADMET tasks when compared to strong reproduced baselines.