Integrating Chemistry Knowledge in Large Language Models via Prompt Engineering

TL;DR

This work tackles the challenge of scarce labeled data in scientific domains by embedding domain-specific knowledge into prompt engineering for large language models. It introduces a domain-knowledge embedded prompt engineering framework that leverages a multi-expert prompt-assembly approach to guide LLMs in chemistry, biology, and materials science tasks, outperforming zero-shot, few-shot, expert, and standard chain-of-thought prompts on a domain-specific benchmark. A 1280-item dataset across organic small molecules, enzymes, and crystal materials supports rigorous evaluation via metrics like Capability, Accuracy, F1, and Hallucination Drop, with numerical tasks transformed to multiple-choice formats for consistent scoring. Case studies on MacMillan’s catalyst, paclitaxel, and LiCoO2 illustrate how domain-aware prompts enable more accurate reasoning and synthesis planning, underscoring the potential of domain-guided prompting to accelerate scientific discovery while highlighting future directions such as broader domain coverage, multi-modal prompts, and human-in-the-loop refinement.

Abstract

This paper presents a study on the integration of domain-specific knowledge in prompt engineering to enhance the performance of large language models (LLMs) in scientific domains. A benchmark dataset is curated to encapsulate the intricate physical-chemical properties of small molecules, their drugability for pharmacology, alongside the functional attributes of enzymes and crystal materials, underscoring the relevance and applicability across biological and chemical domains.The proposed domain-knowledge embedded prompt engineering method outperforms traditional prompt engineering strategies on various metrics, including capability, accuracy, F1 score, and hallucination drop. The effectiveness of the method is demonstrated through case studies on complex materials including the MacMillan catalyst, paclitaxel, and lithium cobalt oxide. The results suggest that domain-knowledge prompts can guide LLMs to generate more accurate and relevant responses, highlighting the potential of LLMs as powerful tools for scientific discovery and innovation when equipped with domain-specific prompts. The study also discusses limitations and future directions for domain-specific prompt engineering development.

Figures (33)

• Figure 1: The Whole Process of Prompt Engineering Framework
• Figure 2: Question Construction, Answer Alignment and Grading Process
• Figure 3: Illustration of the Mainstream Prompt Engineering Methods
• Figure 4: The Whole Process of Domain-Knowledge Prompt Engineering Method
• Figure 5: Capability and Accuracy for All Tasks