Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GEOM-Drugs Revisited: Toward More Chemically Accurate Benchmarks for 3D Molecule Generation

Filipp Nikitin, Ian Dunn, David Ryan Koes, Olexandr Isayev

TL;DR

This work reexamines GEOM-Drugs to identify critical flaws in current evaluation practices, including valency computation bugs, flawed valency lookup tables, and reliance on force-field benchmarks. It introduces a chemically rigorous pipeline with corrected data preprocessing, aromaticity-aware valency tables, and a GFN2-xTB-based geometry/energy benchmark, then retrains multiple leading models on the cleaned dataset. The results show that while absolute scores shift under the corrected framework, relative model rankings largely persist, and diffusion-based methods more accurately reproduce reference geometries than MMFF baselines. The study provides practical recommendations and open-source scripts to enable chemically faithful, interpretable benchmarking of 3D molecular generation.

Abstract

Deep generative models have shown significant promise in generating valid 3D molecular structures, with the GEOM-Drugs dataset serving as a key benchmark. However, current evaluation protocols suffer from critical flaws, including incorrect valency definitions, bugs in bond order calculations, and reliance on force fields inconsistent with the reference data. In this work, we revisit GEOM-Drugs and propose a corrected evaluation framework: we identify and fix issues in data preprocessing, construct chemically accurate valency tables, and introduce a GFN2-xTB-based geometry and energy benchmark. We retrain and re-evaluate several leading models under this framework, providing updated performance metrics and practical recommendations for future benchmarking. Our results underscore the need for chemically rigorous evaluation practices in 3D molecular generation. Our recommended evaluation methods and GEOM-Drugs processing scripts are available at https://github.com/isayevlab/geom-drugs-3dgen-evaluation.

GEOM-Drugs Revisited: Toward More Chemically Accurate Benchmarks for 3D Molecule Generation

TL;DR

This work reexamines GEOM-Drugs to identify critical flaws in current evaluation practices, including valency computation bugs, flawed valency lookup tables, and reliance on force-field benchmarks. It introduces a chemically rigorous pipeline with corrected data preprocessing, aromaticity-aware valency tables, and a GFN2-xTB-based geometry/energy benchmark, then retrains multiple leading models on the cleaned dataset. The results show that while absolute scores shift under the corrected framework, relative model rankings largely persist, and diffusion-based methods more accurately reproduce reference geometries than MMFF baselines. The study provides practical recommendations and open-source scripts to enable chemically faithful, interpretable benchmarking of 3D molecular generation.

Abstract

Deep generative models have shown significant promise in generating valid 3D molecular structures, with the GEOM-Drugs dataset serving as a key benchmark. However, current evaluation protocols suffer from critical flaws, including incorrect valency definitions, bugs in bond order calculations, and reliance on force fields inconsistent with the reference data. In this work, we revisit GEOM-Drugs and propose a corrected evaluation framework: we identify and fix issues in data preprocessing, construct chemically accurate valency tables, and introduce a GFN2-xTB-based geometry and energy benchmark. We retrain and re-evaluate several leading models under this framework, providing updated performance metrics and practical recommendations for future benchmarking. Our results underscore the need for chemically rigorous evaluation practices in 3D molecular generation. Our recommended evaluation methods and GEOM-Drugs processing scripts are available at https://github.com/isayevlab/geom-drugs-3dgen-evaluation.
Paper Structure (18 sections, 3 equations, 3 figures, 5 tables)

This paper contains 18 sections, 3 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Molecule Stability
  3. Identified Issues
  4. Solution
  5. 3D Molecule Evaluation
  6. Challenges in proper and accurate 3D structure assesment
  7. GFN2-xTB energy-based geometry benchmark
  8. Bond Length Differences
  9. Bond Angle Differences
  10. Torsion Angle Differences
  11. Results
  12. Conclusion
  13. Acknowledgement
  14. Appendix I: Valency Lookup Tables for Stability Evaluation
  15. Table \ref{['tab:accurate_valencies']}: Allowed Valencies.
  16. ...and 3 more sections

Figures (3)

  • Figure 1: An example of a molecule where the assumption that aromatic bonds contribute 1.5 to atomic valency holds only partially. In the aromatic form of triphenylene (\ref{['fig:tr_aromat']}), the green-highlighted atoms are correctly classified as stable under the 1.5 assumption, while others are misclassified. In contrast, the kekulized representation (\ref{['fig:tr_kekulized']}) resolves the ambiguity and yields chemically accurate valency assignments across all atoms. This illustrates the limitations of the 1.5 approximation in polycyclic aromatic systems.
  • Figure 2: Examples of molecules that pass the molecular stability test under commonly used evaluation criteria. These flawed metrics erroneously classify chemically invalid configurations as stable—including cases such as a neutral carbon with three single bonds (\ref{['fig:molecule-a']}), a neutral nitrogen with two single bonds (\ref{['fig:molecule-b']}), and a nitrogen atom with +1 charge bonded via both a triple bond and an aromatic bond (\ref{['fig:molecule-c']}).
  • Figure 3: Examples from GEOM-Drugs where GFN2-xTB failed and resulted in fractured molecules. The first row of molecules have neutral carbon with valency 2 and those in the second row have a positively charged hydrogen with valency zero.