Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Re-evaluating Retrosynthesis Algorithms with Syntheseus

Krzysztof Maziarz, Austin Tripp, Guoqing Liu, Megan Stanley, Shufang Xie, Piotr Gaiński, Philipp Seidl, Marwin Segler

TL;DR

This work presents a synthesis planning library with an extensive benchmarking framework, called SYNTHESEUS, which promotes best practice by default, enabling consistent meaningful evaluation of single-step and multi-step synthesis planning algorithms.

Abstract

Automated Synthesis Planning has recently re-emerged as a research area at the intersection of chemistry and machine learning. Despite the appearance of steady progress, we argue that imperfect benchmarks and inconsistent comparisons mask systematic shortcomings of existing techniques, and unnecessarily hamper progress. To remedy this, we present a synthesis planning library with an extensive benchmarking framework, called syntheseus, which promotes best practice by default, enabling consistent meaningful evaluation of single-step models and multi-step planning algorithms. We demonstrate the capabilities of syntheseus by re-evaluating several previous retrosynthesis algorithms, and find that the ranking of state-of-the-art models changes in controlled evaluation experiments. We end with guidance for future works in this area, and call the community to engage in the discussion on how to improve benchmarks for synthesis planning.

Re-evaluating Retrosynthesis Algorithms with Syntheseus

TL;DR

This work presents a synthesis planning library with an extensive benchmarking framework, called SYNTHESEUS, which promotes best practice by default, enabling consistent meaningful evaluation of single-step and multi-step synthesis planning algorithms.

Abstract

Automated Synthesis Planning has recently re-emerged as a research area at the intersection of chemistry and machine learning. Despite the appearance of steady progress, we argue that imperfect benchmarks and inconsistent comparisons mask systematic shortcomings of existing techniques, and unnecessarily hamper progress. To remedy this, we present a synthesis planning library with an extensive benchmarking framework, called syntheseus, which promotes best practice by default, enabling consistent meaningful evaluation of single-step models and multi-step planning algorithms. We demonstrate the capabilities of syntheseus by re-evaluating several previous retrosynthesis algorithms, and find that the ranking of state-of-the-art models changes in controlled evaluation experiments. We end with guidance for future works in this area, and call the community to engage in the discussion on how to improve benchmarks for synthesis planning.
Paper Structure (27 sections, 5 figures, 2 tables)

This paper contains 27 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Prior Work on Benchmarking
  3. Pitfalls and Best Practice for Retrosynthesis Evaluation
  4. Single-Step Models
  5. Multi-Step Search
  6. Syntheseus
  7. Unrestricted single-step model development
  8. Separation of components in multi-step search
  9. Detailed metrics for multi-step search
  10. Experiments: re-evaluation of existing methods
  11. Single-Step
  12. Datasets
  13. Models
  14. Metrics
  15. Setup
  16. ...and 12 more sections

Figures (5)

  • Figure 1: Benchmarking workflows and metrics studied in this work.
  • Figure 2: Trade-off between top-5 accuracy and inference speed. Circle area is proportional to the number of parameters; color denotes whether a model uses reaction templates (blue), generates a sequence of graph edits (green) or produces the output SMILES from scratch (red). Dashed gray line shows the Pareto front (best result for any time budget). Exact results for Chemformer are not shown as they fall below the plot boundary. We show in-distribution results on USPTO-50K (left) and out-of-distribution generalization on Pistachio (right).
  • Figure 3: Multi-step search results on the Retro* Hard target set with different single-step models. Left: Time until first solution was found (or $\emptyset$ if a molecule was not solved). Orange line represents the median, box represents 25th and 75th percentile, whiskers represent 5th and 95th percentile, points outside this range are shown as dots. Right: Approximate number of non-overlapping routes present in the search graph (tracked over time and aggregated across target molecules). Solid line represents the median, shaded area shows the 40th and 60th percentile. On the right hand side we note the average number of calls made by the model within the allotted time limit.
  • Figure 4: Results on USPTO-50K in same format as Figure \ref{['fig:single-step']} but extended with top-1, top-3, top-10, top-50, and MRR. Plot for top-5 is reprinted here for convenience.
  • Figure 5: Results on Pistachio in same format as Figure \ref{['fig:single-step']} but extended with top-1, top-3, top-10, top-50, and MRR. Plot for top-5 is reprinted here for convenience.