Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs

Bogdan Zagribelnyy, Ivan Ilin, Maksim Kuznetsov, Nikita Bondarev, Roman Schutski, Thomas MacDougall, Rim Shayakhmetov, Zulfat Miftakhutdinov, Mikolaj Mizera, Vladimir Aladinskiy, Alex Aliper, Alex Zhavoronkov

TL;DR

This work critiques the reliance on ground-truth, Top-K metrics for single-step retrosynthesis and proposes a plausibility-focused benchmarking framework. It introduces ChemCensor, a precedent-based metric that scores reaction steps by reaction centers and functional-group contexts, and CREED, a large-scale, ChemCensor-verified reaction dataset, to train and evaluate LLMs. The URSA-expert-2026 benchmark provides a challenging out-of-domain testbed, and results show that models trained on CREED outperform general-purpose and chemistry-specialist baselines. The authors also demonstrate reinforcement-learning fine-tuning with ChemCensor rewards improves maximum plausibility scores, highlighting the value of legality-guided feedback for retrosynthesis. Overall, the framework aims to improve reproducibility and alignment with human planning in drug synthesis by prioritizing chemical plausibility over exact-match accuracy.

Abstract

Recent progress has expanded the use of large language models (LLMs) in drug discovery, including synthesis planning. However, objective evaluation of retrosynthesis performance remains limited. Existing benchmarks and metrics typically rely on published synthetic procedures and Top-K accuracy based on single ground-truth, which does not capture the open-ended nature of real-world synthesis planning. We propose a new benchmarking framework for single-step retrosynthesis that evaluates both general-purpose and chemistry-specialized LLMs using ChemCensor, a novel metric for chemical plausibility. By emphasizing plausibility over exact match, this approach better aligns with human synthesis planning practices. We also introduce CREED, a novel dataset comprising millions of ChemCensor-validated reaction records for LLM training, and use it to train a model that improves over the LLM baselines under this benchmark.

When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs

TL;DR

This work critiques the reliance on ground-truth, Top-K metrics for single-step retrosynthesis and proposes a plausibility-focused benchmarking framework. It introduces ChemCensor, a precedent-based metric that scores reaction steps by reaction centers and functional-group contexts, and CREED, a large-scale, ChemCensor-verified reaction dataset, to train and evaluate LLMs. The URSA-expert-2026 benchmark provides a challenging out-of-domain testbed, and results show that models trained on CREED outperform general-purpose and chemistry-specialist baselines. The authors also demonstrate reinforcement-learning fine-tuning with ChemCensor rewards improves maximum plausibility scores, highlighting the value of legality-guided feedback for retrosynthesis. Overall, the framework aims to improve reproducibility and alignment with human planning in drug synthesis by prioritizing chemical plausibility over exact-match accuracy.

Abstract

Recent progress has expanded the use of large language models (LLMs) in drug discovery, including synthesis planning. However, objective evaluation of retrosynthesis performance remains limited. Existing benchmarks and metrics typically rely on published synthetic procedures and Top-K accuracy based on single ground-truth, which does not capture the open-ended nature of real-world synthesis planning. We propose a new benchmarking framework for single-step retrosynthesis that evaluates both general-purpose and chemistry-specialized LLMs using ChemCensor, a novel metric for chemical plausibility. By emphasizing plausibility over exact match, this approach better aligns with human synthesis planning practices. We also introduce CREED, a novel dataset comprising millions of ChemCensor-validated reaction records for LLM training, and use it to train a model that improves over the LLM baselines under this benchmark.
Paper Structure (34 sections, 3 equations, 20 figures, 4 tables)

This paper contains 34 sections, 3 equations, 20 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmarking solutions for SSRS
  4. LLMs for the SSRS task
  5. Approach
  6. ChemCensor Metric
  7. Benchmarking Sets
  8. Aggregated Metrics
  9. CREED
  10. Experiments
  11. Baseline Models
  12. C3LM Supervised Fine-Tuning
  13. C3LM Reinforcement Learning Fine-Tuning
  14. Training procedure:
  15. ChemCensor Reward:
  16. ...and 19 more sections

Figures (20)

  • Figure 1: Assessment of generated reactants from benchmarked models by ChemCensor; each answer is assigned to pass or specific non-pass categories, and models are ordered by the fraction of ChemCensor passed reactions.
  • Figure 2: ChemCensor part of the URSA benchmark. A. The procedure of retrieval and chemistry-aware knowledge aggregation from USPTO-full into the relational DB of RCs and FGs. B. The pipeline of LLMs benchmarking for the SSRS task. C. The example of RC and FGs intercompatibility assessment during single reaction scoring within ChemCensor.
  • Figure 3: CREED (Comprehensive Reactant Exhaustive Enumeration Dataset) design process.
  • Figure 4: LLMs' performance on URSA-expert-2026 set.
  • Figure 5: FG–related error profiles across models on test sets. The heat map shows the count of CC FG incompatibility errors across the best $10$ models by Av. PT-Top-10 CC (see \ref{['tab:chemcensor_benchmark_results']}), darker cells indicate a higher error number. See FGs in \ref{['app:fg_smarts']}.
  • ...and 15 more figures