Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning

Mateusz Praski, Jakub Adamczyk, Wojciech Czech

TL;DR

This study interrogates the assumed progress in pretrained molecular embeddings by benchmarking 25 models across 25 diverse datasets using a fair, fixed-embedding protocol and a Bayesian Bradley-Terry framework. The analysis reveals that, aside from the CLAMP fingerprint-based model, most neural representations fail to outperform the strong baseline ECFP fingerprints, and several transformer-based approaches offer only practical equivalence with substantial computational cost. By enforcing rigorous evaluation and uncertainty-aware ranking via ROPE, the work highlights the need for domain-specific inductive biases and robust baselines in molecular representation learning. The findings have practical implications for model selection in drug discovery and chemoinformatics, urging researchers to prioritize reproducible benchmarks and cost-efficient baselines while continuing to refine representation learning with targeted, chemistry-informed improvements.

Abstract

Pretrained neural networks have attracted significant interest in chemistry and small molecule drug design. Embeddings from these models are widely used for molecular property prediction, virtual screening, and small data learning in molecular chemistry. This study presents the most extensive comparison of such models to date, evaluating 25 models across 25 datasets. Under a fair comparison framework, we assess models spanning various modalities, architectures, and pretraining strategies. Using a dedicated hierarchical Bayesian statistical testing model, we arrive at a surprising result: nearly all neural models show negligible or no improvement over the baseline ECFP molecular fingerprint. Only the CLAMP model, which is also based on molecular fingerprints, performs statistically significantly better than the alternatives. These findings raise concerns about the evaluation rigor in existing studies. We discuss potential causes, propose solutions, and offer practical recommendations.

Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning

TL;DR

This study interrogates the assumed progress in pretrained molecular embeddings by benchmarking 25 models across 25 diverse datasets using a fair, fixed-embedding protocol and a Bayesian Bradley-Terry framework. The analysis reveals that, aside from the CLAMP fingerprint-based model, most neural representations fail to outperform the strong baseline ECFP fingerprints, and several transformer-based approaches offer only practical equivalence with substantial computational cost. By enforcing rigorous evaluation and uncertainty-aware ranking via ROPE, the work highlights the need for domain-specific inductive biases and robust baselines in molecular representation learning. The findings have practical implications for model selection in drug discovery and chemoinformatics, urging researchers to prioritize reproducible benchmarks and cost-efficient baselines while continuing to refine representation learning with targeted, chemistry-informed improvements.

Abstract

Pretrained neural networks have attracted significant interest in chemistry and small molecule drug design. Embeddings from these models are widely used for molecular property prediction, virtual screening, and small data learning in molecular chemistry. This study presents the most extensive comparison of such models to date, evaluating 25 models across 25 datasets. Under a fair comparison framework, we assess models spanning various modalities, architectures, and pretraining strategies. Using a dedicated hierarchical Bayesian statistical testing model, we arrive at a surprising result: nearly all neural models show negligible or no improvement over the baseline ECFP molecular fingerprint. Only the CLAMP model, which is also based on molecular fingerprints, performs statistically significantly better than the alternatives. These findings raise concerns about the evaluation rigor in existing studies. We discuss potential causes, propose solutions, and offer practical recommendations.

Paper Structure

This paper contains 35 sections, 6 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Molecular Embedding Approaches
  3. Molecular fingerprints
  4. Graph neural networks
  5. Graph transformers
  6. Text transformers
  7. Hybrid models
  8. Methods
  9. Bayesian Bradley-Terry model
  10. Likelihood
  11. Priors
  12. Posterior inference
  13. Posterior summaries
  14. Decision rule and model ranking
  15. Step-by-step protocol
  16. ...and 20 more sections

Figures (3)

  • Figure 1: The proposed workflow.
  • Figure 2: Pairwise win ratio between all evaluated models. Green marks wins of model on Y axis against model on X axis, red means losses (i.e., model from X axis won more).
  • Figure 3: Percentage of models outperforming ECFP for each dataset.