Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Developing ChemDFM as a large language foundation model for chemistry

Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Yi Xia, Bo Chen, Hongshen Xu, Zichen Zhu, Su Zhu, Shuai Fan, Guodong Shen, Kai Yu, Xin Chen

TL;DR

ChemDFM develops a chemistry-focused large language model by two-stage specialization of a general-domain LLM: domain pre-training on an extensive chemistry corpus to embed domain knowledge, followed by instruction tuning with chemistry-centric prompts and notations. It demonstrates strong performance across molecule recognition, design, property prediction, and reaction understanding, often outperforming open-source LLMs and approaching GPT-4. The work also showcases free-form, dialogue-based collaboration capabilities, positioning ChemDFM as a practical AI assistant for chemical research. Open-sourced weights, datasets, and evaluation tools further enable reuse and benchmarking in the field of chemistry. Overall, ChemDFM represents a significant step toward integrated chemical knowledge and language abilities in a single foundation model for science.

Abstract

Artificial intelligence (AI) has played an increasingly important role in chemical research. However, most models currently used in chemistry are specialist models that require training and tuning for specific tasks. A more generic and efficient solution would be an AI model that could address many tasks and support free-form dialogue in the broad field of chemistry. In its utmost form, such a generalist AI chemist could be referred to as Chemical General Intelligence. Large language models (LLMs) have recently logged tremendous success in the general domain of natural language processing, showing emerging task generalization and free-form dialogue capabilities. However, domain knowledge of chemistry is largely missing when training general-domain LLMs. The lack of such knowledge greatly hinders the performance of generalist LLMs in the field of chemistry. To this end, we develop ChemDFM, a pioneering LLM for chemistry trained on 34B tokens from chemical literature and textbooks, and fine-tuned using 2.7M instructions. As a result, it can understand and reason with chemical knowledge in free-form dialogue. Quantitative evaluations show that ChemDFM significantly surpasses most representative open-source LLMs. It outperforms GPT-4 on a great portion of chemical tasks, despite the substantial size difference. We have open-sourced the inference codes, evaluation datasets, and model weights of ChemDFM on Huggingface (https://huggingface.co/OpenDFM/ChemDFM-v1.0-13B).

Developing ChemDFM as a large language foundation model for chemistry

TL;DR

ChemDFM develops a chemistry-focused large language model by two-stage specialization of a general-domain LLM: domain pre-training on an extensive chemistry corpus to embed domain knowledge, followed by instruction tuning with chemistry-centric prompts and notations. It demonstrates strong performance across molecule recognition, design, property prediction, and reaction understanding, often outperforming open-source LLMs and approaching GPT-4. The work also showcases free-form, dialogue-based collaboration capabilities, positioning ChemDFM as a practical AI assistant for chemical research. Open-sourced weights, datasets, and evaluation tools further enable reuse and benchmarking in the field of chemistry. Overall, ChemDFM represents a significant step toward integrated chemical knowledge and language abilities in a single foundation model for science.

Abstract

Artificial intelligence (AI) has played an increasingly important role in chemical research. However, most models currently used in chemistry are specialist models that require training and tuning for specific tasks. A more generic and efficient solution would be an AI model that could address many tasks and support free-form dialogue in the broad field of chemistry. In its utmost form, such a generalist AI chemist could be referred to as Chemical General Intelligence. Large language models (LLMs) have recently logged tremendous success in the general domain of natural language processing, showing emerging task generalization and free-form dialogue capabilities. However, domain knowledge of chemistry is largely missing when training general-domain LLMs. The lack of such knowledge greatly hinders the performance of generalist LLMs in the field of chemistry. To this end, we develop ChemDFM, a pioneering LLM for chemistry trained on 34B tokens from chemical literature and textbooks, and fine-tuned using 2.7M instructions. As a result, it can understand and reason with chemical knowledge in free-form dialogue. Quantitative evaluations show that ChemDFM significantly surpasses most representative open-source LLMs. It outperforms GPT-4 on a great portion of chemical tasks, despite the substantial size difference. We have open-sourced the inference codes, evaluation datasets, and model weights of ChemDFM on Huggingface (https://huggingface.co/OpenDFM/ChemDFM-v1.0-13B).
Paper Structure (35 sections, 1 equation, 12 figures, 12 tables)

This paper contains 35 sections, 1 equation, 12 figures, 12 tables.

Table of Contents

  1. Introduction
  2. ChemDFM
  3. Domain Pre-training
  4. Instruction Tuning
  5. Evaluations
  6. Quantitative Evaluation
  7. 1) Molecule recognition.
  8. 2) Text-based molecule design.
  9. 3) Molecular property prediction.
  10. 4) Reaction prediction and retrosynthesis.
  11. LLM-Based Research Assistant
  12. Related Work
  13. Conclusion
  14. Experimental Setups
  15. Domain Pre-training
  16. ...and 20 more sections

Figures (12)

  • Figure 1: Scheme to obtain an LLM for chemistry, through using chemical domain knowledge to train a general-domain LLM.
  • Figure 2: a) Two-step training procedure to obtain ChemDFM. The icons are generated by the SDXL model provided by Stability AI. b) Various types of tasks ChemDFM is capable of.
  • Figure 3: Representative question used in instruction tuning.
  • Figure 4: Examples of paper reading. Answers from ChemDFM are compared with GPT and the base model LLaMa. Correct and relevant information in the replies is marked in green, correct but irrelevant information in yellow, and wrong information in red. Key points of the answer are marked in bold. Full details and more examples are elaborated in Appendix Section \ref{['sec:pr']}.
  • Figure 5: Example showing ChemDFM as an assistant researcher in the design of experiment through free-form dialogue. Key points of the answer are marked in bold. More examples can be found in Appendix Section \ref{['sec:dbc']}.
  • ...and 7 more figures