Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-scale Graph Autoregressive Modeling: Molecular Property Prediction via Next Token Prediction

Zhuoyang Jiang, Yaosen Min, Peiran Jin, Lei Chen

TL;DR

This work tackles molecular property prediction with decoder-only Transformers by bridging topology-exposure gaps in SMILES-based approaches and the corruption risks of graph masking. It introduces CamS, a three-stage graph-to-sequence interface that mines data-driven motifs, serializes them with scaffold-rooted BFS, and stacks multi-scale subsequences from fine to coarse to enable standard next-token prediction without architectural changes. The CamS-LLaMA pipeline pre-trains a vanilla decoder on CamS sequences and applies a lightweight fingerprint prior only during fine-tuning, achieving state-of-the-art results on MoleculeNet and MoleculeACE while offering mechanistic interpretability: multi-scale causal serialization directs attention to cliff-driving structural differences. Overall, CamS demonstrates that with an appropriate graph-to-sequence representation, vanilla autoregressive foundation models can effectively learn topology-rich molecular representations at scale, enabling both high predictive performance and transparent reasoning about activity cliffs.

Abstract

We present Connection-Aware Motif Sequencing (CamS), a graph-to-sequence representation that enables decoder-only Transformers to learn molecular graphs via standard next-token prediction (NTP). For molecular property prediction, SMILES-based NTP scales well but lacks explicit topology, whereas graph-native masked modeling captures connectivity but risks disrupting the pivotal chemical details (e.g., activity cliffs). CamS bridges this gap by serializing molecular graphs into structure-rich causal sequences. CamS first mines data-driven connection-aware motifs. It then serializes motifs via scaffold-rooted breadth-first search (BFS) to establish a stable core-to-periphery order. Crucially, CamS enables hierarchical modeling by concatenating sequences from fine to coarse motif scales, allowing the model to condition global scaffolds on dense, uncorrupted local structural evidence. We instantiate CamS-LLaMA by pre-training a vanilla LLaMA backbone on CamS sequences. It achieves state-of-the-art performance on MoleculeNet and the activity-cliff benchmark MoleculeACE, outperforming both SMILES-based language models and strong graph baselines. Interpretability analysis confirms that our multi-scale causal serialization effectively drives attention toward cliff-determining differences.

Multi-scale Graph Autoregressive Modeling: Molecular Property Prediction via Next Token Prediction

TL;DR

This work tackles molecular property prediction with decoder-only Transformers by bridging topology-exposure gaps in SMILES-based approaches and the corruption risks of graph masking. It introduces CamS, a three-stage graph-to-sequence interface that mines data-driven motifs, serializes them with scaffold-rooted BFS, and stacks multi-scale subsequences from fine to coarse to enable standard next-token prediction without architectural changes. The CamS-LLaMA pipeline pre-trains a vanilla decoder on CamS sequences and applies a lightweight fingerprint prior only during fine-tuning, achieving state-of-the-art results on MoleculeNet and MoleculeACE while offering mechanistic interpretability: multi-scale causal serialization directs attention to cliff-driving structural differences. Overall, CamS demonstrates that with an appropriate graph-to-sequence representation, vanilla autoregressive foundation models can effectively learn topology-rich molecular representations at scale, enabling both high predictive performance and transparent reasoning about activity cliffs.

Abstract

We present Connection-Aware Motif Sequencing (CamS), a graph-to-sequence representation that enables decoder-only Transformers to learn molecular graphs via standard next-token prediction (NTP). For molecular property prediction, SMILES-based NTP scales well but lacks explicit topology, whereas graph-native masked modeling captures connectivity but risks disrupting the pivotal chemical details (e.g., activity cliffs). CamS bridges this gap by serializing molecular graphs into structure-rich causal sequences. CamS first mines data-driven connection-aware motifs. It then serializes motifs via scaffold-rooted breadth-first search (BFS) to establish a stable core-to-periphery order. Crucially, CamS enables hierarchical modeling by concatenating sequences from fine to coarse motif scales, allowing the model to condition global scaffolds on dense, uncorrupted local structural evidence. We instantiate CamS-LLaMA by pre-training a vanilla LLaMA backbone on CamS sequences. It achieves state-of-the-art performance on MoleculeNet and the activity-cliff benchmark MoleculeACE, outperforming both SMILES-based language models and strong graph baselines. Interpretability analysis confirms that our multi-scale causal serialization effectively drives attention toward cliff-determining differences.
Paper Structure (73 sections, 1 theorem, 21 equations, 1 figure, 9 tables, 7 algorithms)

This paper contains 73 sections, 1 theorem, 21 equations, 1 figure, 9 tables, 7 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Molecular Property Prediction within Foundation Models Paradigm
  4. Molecular Tokenization Strategies
  5. Method
  6. CamS-Tokenizer
  7. Substructure Level: CamS Vocabulary
  8. Per-Scale Encoding
  9. Cross-Scale Concatenation
  10. CamS-LLaMA
  11. Autoregressive Pre-training via NTP
  12. Fine-tuning with Fingerprint Injection
  13. CamS Representation vs. Graph Transformer
  14. Unified Framework.
  15. Information-Theoretic Context Analysis.
  16. ...and 58 more sections

Key Result

Proposition 3.1

For any non-trivial masking channel $\mathcal{M}$, we have

Figures (1)

  • Figure 1: The overall framework of CamS-LLaMA.(a) CamS-Tokenizer: Molecules are transformed into multi-scale causal sequences. Through Per-scale Encoding, motif graphs are serialized via Scaffold-rooted BFS (Intra-Scale Order) to establish CamS Subsequence, followed by Cross-Scale Concatenation to arrange subsequences into a final CamS Sequence from fine to coarse granularities (Inter-Scale Order). (b) Pre-training Pipeline: The model is pre-trained on CamS Token Lists using a standard LLaMA via NTP, learning to aggregate structural information. (c) Fine-tuning Pipeline: During downstream tasks, the pre-trained weights are transferred, and a lightweight fingerprint prior is injected (via projection, prepending in embedding layer and fusion in classifier or regressor) to take part in the prediction of specific properties.

Theorems & Definitions (2)

  • Proposition 3.1: Context Information Inequality
  • proof