Unveiling Scaling Behaviors in Molecular Language Models: Effects of Model Size, Data, and Representation

TL;DR

This work systematically characterizes scaling in molecular language models under compute budgets by exhaustively varying model size, data, and molecular representations across 300 models and 10,000 experiments. It establishes a bivariate power-law for validation loss, derives a compute-optimal frontier with closed-form expressions for the optimal size $P_{ ext{opt}}(C)$ and tokens $D_{ ext{opt}}(C)$, and shows that representation choice shifts the frontier and irreducible loss $L_{ inf}$. Downstream evaluation on MoleculeNet demonstrates task- and representation-dependent scaling behavior, with fragment-based representations (FragSeq/FragLink) often yielding stronger or more robust gains, particularly for biophysics and certain classification tasks. The study provides practical guidance for allocating compute between model capacity and data, highlights the importance of representation design, and publicly releases the largest library of molecular language models to date to accelerate future research.

Abstract

Molecular generative models, often employing GPT-style language modeling on molecular string representations, have shown promising capabilities when scaled to large datasets and model sizes. However, it remains unclear and subject to debate whether these models adhere to predictable scaling laws under fixed computational budgets, which is a crucial understanding for optimally allocating resources between model size, data volume, and molecular representation. In this study, we systematically investigate the scaling behavior of molecular language models across both pretraining and downstream tasks. We train 300 models and conduct over 10,000 experiments, rigorously controlling compute budgets while independently varying model size, number of training tokens, and molecular representation. Our results demonstrate clear scaling laws in molecular models for both pretraining and downstream transfer, reveal the substantial impact of molecular representation on performance, and explain previously observed inconsistencies in scaling behavior for molecular generation. Additionally, we publicly release the largest library of molecular language models to date to facilitate future research and development. Code and models are available at https://github.com/SZU-ADDG/MLM-Scaling.

Figures (21)

• Figure 1: Pretraining loss scaling under compute-controlled analysis. Points are end-of-run validation losses from single-epoch from-scratch runs. The shaded region marks the compute range covered by our training grid, and dashed segments show extrapolation beyond this range.
• Figure 2: An overview of our research framework. (a) Data & Representations: Beginning with raw molecular data from the ZINC and UniChem databases, each molecule is converted into five distinct string-based representations: DeepSMILES, FragLink, FragSeq, SAFE and SMILES. (b) Model Architecture: A GPT-based model is used for all experiments. The pre-training phase utilizes an autoregressive prediction objective to train models of varying sizes (from 1M to 650M parameters) on different data scales (from 100M to 3B tokens). The fine-tuning phase adapts the pre-trained model using LoRA for specific downstream regression or classification tasks. (c) Evaluation: The model's capabilities are assessed across a wide range of tasks, including the predicted minimal validation loss along the compute-optimal frontier and a comprehensive suite of property prediction benchmarks spanning biochemistry, physiology and biophysics.
• Figure 3: Compute-controlled views from the fitted bivariate law. (a--e) IsoFLOP curves under fixed compute budgets, with observed single-epoch runs overlaid. (f--j) IsoLoss contours in the $(C,P)$ plane, with the compute-optimal frontier highlighted.
• Figure 4: Longer training on a fixed corpus. Each panel reports end-of-epoch validation loss versus compute for repeated passes. See Appendix Figure \ref{['fig:longer_more']} for details.
• Figure 5: SMILES, $P{=}1$M, $B{=}100$M: end-of-epoch validation loss from epoch 1 to 10.