Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Grounding Large Language Models in Reaction Knowledge Graphs for Synthesis Retrieval

Olga Bunkova, Lorenzo Di Fruscia, Sophia Rupprecht, Artur M. Schweidtmann, Marcel J. T. Reinders, Jana M. Weber

TL;DR

This work tackles hallucination and knowledge drift in LLM-assisted synthesis planning by grounding reasoning in a reaction knowledge graph and translating natural language queries into executable Cypher via Text2Cypher. It constructs a bipartite KG from USPTO reactions, defines single- and multi-step retrosynthesis tasks, and systematically studies zero-shot and one-shot prompting with static, random, and embedding-based exemplars alongside a Chain-of-Verification correction loop. Key findings show that moving from zero- to one-shot prompting yields the largest gains, especially for multi-step routes, and that text-similarity metrics poorly predict retrieval success due to semantic and structural variability in Cypher. The paper offers a reproducible evaluation setup and design recommendations, emphasizing task-aligned exemplars and task-specific validators to advance KG-grounded synthesis retrieval, with code available for broader adoption.$

Abstract

Large Language Models (LLMs) can aid synthesis planning in chemistry, but standard prompting methods often yield hallucinated or outdated suggestions. We study LLM interactions with a reaction knowledge graph by casting reaction path retrieval as a Text2Cypher (natural language to graph query) generation problem, and define single- and multi-step retrieval tasks. We compare zero-shot prompting to one-shot variants using static, random, and embedding-based exemplar selection, and assess a checklist-driven validator/corrector loop. To evaluate our framework, we consider query validity and retrieval accuracy. We find that one-shot prompting with aligned exemplars consistently performs best. Our checklist-style self-correction loop mainly improves executability in zero-shot settings and offers limited additional retrieval gains once a good exemplar is present. We provide a reproducible Text2Cypher evaluation setup to facilitate further work on KG-grounded LLMs for synthesis planning. Code is available at https://github.com/Intelligent-molecular-systems/KG-LLM-Synthesis-Retrieval.

Grounding Large Language Models in Reaction Knowledge Graphs for Synthesis Retrieval

TL;DR

This work tackles hallucination and knowledge drift in LLM-assisted synthesis planning by grounding reasoning in a reaction knowledge graph and translating natural language queries into executable Cypher via Text2Cypher. It constructs a bipartite KG from USPTO reactions, defines single- and multi-step retrosynthesis tasks, and systematically studies zero-shot and one-shot prompting with static, random, and embedding-based exemplars alongside a Chain-of-Verification correction loop. Key findings show that moving from zero- to one-shot prompting yields the largest gains, especially for multi-step routes, and that text-similarity metrics poorly predict retrieval success due to semantic and structural variability in Cypher. The paper offers a reproducible evaluation setup and design recommendations, emphasizing task-aligned exemplars and task-specific validators to advance KG-grounded synthesis retrieval, with code available for broader adoption.$

Abstract

Large Language Models (LLMs) can aid synthesis planning in chemistry, but standard prompting methods often yield hallucinated or outdated suggestions. We study LLM interactions with a reaction knowledge graph by casting reaction path retrieval as a Text2Cypher (natural language to graph query) generation problem, and define single- and multi-step retrieval tasks. We compare zero-shot prompting to one-shot variants using static, random, and embedding-based exemplar selection, and assess a checklist-driven validator/corrector loop. To evaluate our framework, we consider query validity and retrieval accuracy. We find that one-shot prompting with aligned exemplars consistently performs best. Our checklist-style self-correction loop mainly improves executability in zero-shot settings and offers limited additional retrieval gains once a good exemplar is present. We provide a reproducible Text2Cypher evaluation setup to facilitate further work on KG-grounded LLMs for synthesis planning. Code is available at https://github.com/Intelligent-molecular-systems/KG-LLM-Synthesis-Retrieval.
Paper Structure (24 sections, 12 figures)

This paper contains 24 sections, 12 figures.

Table of Contents

  1. Introduction
  2. Methods
  3. Reaction Knowledge Graph: Data and construction
  4. Tasks: Single- and Multi-step retrieval
  5. Text2Cypher prompting and exemplar selection
  6. Checklist validator/corrector
  7. Framework evaluation and implementation
  8. Results
  9. Text-to-text similarity is a poor proxy for retrieval accuracy
  10. Largest performance gains from zero- to one-shot in multi-step tasks
  11. Checklist self-correction (CoVe) has limited benefits, validator is the bottleneck
  12. Conclusions
  13. Appendix
  14. Prompt strategies
  15. Static one-shot prompting
  16. ...and 9 more sections

Figures (12)

  • Figure 1: (A) Chemical reaction equation with reactants and products (left), and a possible directed bipartite graph representation, where nodes correspond to reactions and molecules (right). (B) Schema used for the reaction knowledge graph. (C, D) Retrosynthesis task types for single- and multi-step reaction retrieval, along with natural language queries and number of task samples. For multi-step reaction retrieval $n \in\{2,3,4\}$.
  • Figure 2: (A) Based on the user question, with (one-shot) or without (zero-shot) an exemplar, an LLM generates a Cypher query with the goal of retrieving relevant reactions/pathways. The query is then executed on the BG in Neo4j. (B) CoVE-style prompting. First, an LLM generates a candidate query. A validator LLM checks it against a fixed checklist, and outputs a list of specific errors if applicable. A corrector LLM then applies minimal edits only to resolve the flagged issues. The process repeats until either the query becomes valid or a maximum of three correction attempts are reached. (C) Format of gold answers for single-step and multi-step retrieval tasks.
  • Figure 3: Text-to-text similarity (for Cypher queries, top row) and retrieval metrics (bottom row) for the single-step task Best-Yielding Reaction under different prompt settings: Zero-shot (ZS), one-shot static (1S-S), one-shot random (1S-D-R), and one-shot semantic (1S-D-S). Within each setting, five prompt versions (indicated in the bottom legend) add increasing contextual/structural guidance.
  • Figure 4: Text-to-text similarity (for Cypher queries, top, left) and retrieval metrics (top, right) for the multi-step task Intermediate Molecule Identification under zero-shot (ZS) and one-shot semantic (1S-D-S). One-shot largely removes zero-shot failures (anchoring, traversal direction, pathway length). Within each setting, five prompt versions add increasing contextual/structural guidance. The error taxonomy table (bottom) reports the rate of non-executable queries, incorrect retrievals, and their absolute counts across five error categories for all multi-step tasks, which cover almost all observed cases.
  • Figure A1: Example banks for single-step (left) and multi-step (right) retrieval. Alongside each task we report their natural language queries and corresponding logical intent.
  • ...and 7 more figures