Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Retrieval-Augmented Generation for Chemistry

Xianrui Zhong, Bowen Jin, Siru Ouyang, Yanzhen Shen, Qiao Jin, Yin Fang, Zhiyong Lu, Jiawei Han

TL;DR

This work introduces ChemRAG-Bench, a 1,932-question chemistry benchmark, and ChemRAG-Toolkit, a modular framework enabling five retrievers and eight LLMs to evaluate retrieval-augmented generation in chemistry. Across diverse tasks, RAG yields an average relative improvement of 17.4% over direct inference, with larger models deriving the most consistent gains. The study provides a detailed analysis of corpus choices, retriever architectures, and retrieval depth, offering practical recommendations for task-aware corpus selection, hybrid retrievers, and deployment considerations in chemistry. The resources aim to standardize evaluation and accelerate progress in chemistry-focused RAG systems, with implications for grounding, up-to-date knowledge, and reduced hallucinations in domain-specific AI systems.

Abstract

Retrieval-augmented generation (RAG) has emerged as a powerful framework for enhancing large language models (LLMs) with external knowledge, particularly in scientific domains that demand specialized and dynamic information. Despite its promise, the application of RAG in the chemistry domain remains underexplored, primarily due to the lack of high-quality, domain-specific corpora and well-curated evaluation benchmarks. In this work, we introduce ChemRAG-Bench, a comprehensive benchmark designed to systematically assess the effectiveness of RAG across a diverse set of chemistry-related tasks. The accompanying chemistry corpus integrates heterogeneous knowledge sources, including scientific literature, the PubChem database, PubMed abstracts, textbooks, and Wikipedia entries. In addition, we present ChemRAG-Toolkit, a modular and extensible RAG toolkit that supports five retrieval algorithms and eight LLMs. Using ChemRAG-Toolkit, we demonstrate that RAG yields a substantial performance gain -- achieving an average relative improvement of 17.4% over direct inference methods. We further conduct in-depth analyses on retriever architectures, corpus selection, and the number of retrieved passages, culminating in practical recommendations to guide future research and deployment of RAG systems in the chemistry domain. The code and data is available at https://chemrag.github.io.

Benchmarking Retrieval-Augmented Generation for Chemistry

TL;DR

This work introduces ChemRAG-Bench, a 1,932-question chemistry benchmark, and ChemRAG-Toolkit, a modular framework enabling five retrievers and eight LLMs to evaluate retrieval-augmented generation in chemistry. Across diverse tasks, RAG yields an average relative improvement of 17.4% over direct inference, with larger models deriving the most consistent gains. The study provides a detailed analysis of corpus choices, retriever architectures, and retrieval depth, offering practical recommendations for task-aware corpus selection, hybrid retrievers, and deployment considerations in chemistry. The resources aim to standardize evaluation and accelerate progress in chemistry-focused RAG systems, with implications for grounding, up-to-date knowledge, and reduced hallucinations in domain-specific AI systems.

Abstract

Retrieval-augmented generation (RAG) has emerged as a powerful framework for enhancing large language models (LLMs) with external knowledge, particularly in scientific domains that demand specialized and dynamic information. Despite its promise, the application of RAG in the chemistry domain remains underexplored, primarily due to the lack of high-quality, domain-specific corpora and well-curated evaluation benchmarks. In this work, we introduce ChemRAG-Bench, a comprehensive benchmark designed to systematically assess the effectiveness of RAG across a diverse set of chemistry-related tasks. The accompanying chemistry corpus integrates heterogeneous knowledge sources, including scientific literature, the PubChem database, PubMed abstracts, textbooks, and Wikipedia entries. In addition, we present ChemRAG-Toolkit, a modular and extensible RAG toolkit that supports five retrieval algorithms and eight LLMs. Using ChemRAG-Toolkit, we demonstrate that RAG yields a substantial performance gain -- achieving an average relative improvement of 17.4% over direct inference methods. We further conduct in-depth analyses on retriever architectures, corpus selection, and the number of retrieved passages, culminating in practical recommendations to guide future research and deployment of RAG systems in the chemistry domain. The code and data is available at https://chemrag.github.io.
Paper Structure (31 sections, 4 figures, 19 tables)

This paper contains 31 sections, 4 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Retrieval-Augmented Generation
  4. Large Language Models for Chemistry
  5. The ChemRAG-Bench Benchmark
  6. Evaluation Settings
  7. Question Datasets
  8. Metric
  9. The ChemRAG-Toolkit
  10. Corpora
  11. Retrievers
  12. LLMs
  13. Experiment Result
  14. Comparison of Backbone LLMs
  15. Comparison of Retrievers and Corpora
  16. ...and 16 more sections

Figures (4)

  • Figure 1: Overview of the ChemRAG toolkit. Retrievers are first constructed from ChemRAG corpora. For each question in the ChemRAG-Bench benchmark, we retrieve related documents as additional contexts for LLMs to predict the final answer.
  • Figure 2: Performance comparison on description-guided molecule design w.r.t evaluation metrics for molecule generation. Ours outperforms the baseline in almost all the scenarios.
  • Figure 3: Performance comparison on different numbers of retrieved documents. The red dotted line represents the baseline. The experiments are conducted on GPT-3.5-turbo.
  • Figure 4: The overall corpus composition of ChemRAG corpora and the actually retrieved proportion in different tasks.