Structured Chemistry Reasoning with Large Language Models

TL;DR

The paper tackles the challenge that large language models struggle with complex chemistry reasoning due to gaps in structured reasoning. It introduces StructChem, a three-phase prompting framework comprising formulae generation, grounded step-by-step reasoning, and confidence-based review/refinement, leveraging program-of-thought style calculations. Across four SciBench datasets and with GPT-4 (andGPT-3.5), StructChem yields substantial accuracy gains in both zero-shot and few-shot settings and enables effective fine-tuning of smaller models using its reasoning data. Through ablations and error analyses, the work demonstrates where the gains come from and highlights remaining challenges in precise, grounded scientific reasoning, suggesting paths toward retrieval-enhanced and knowledge-distilling approaches for broader impact.

Abstract

Large Language Models (LLMs) excel in diverse areas, yet struggle with complex scientific reasoning, especially in the field of chemistry. Different from the simple chemistry tasks (e.g., molecule classification) addressed in previous studies, complex chemistry problems require not only vast knowledge and precise calculation, but also compositional reasoning about rich dynamic interactions of different concepts (e.g., temperature changes). Our study shows that even advanced LLMs, like GPT-4, can fail easily in different ways. Interestingly, the errors often stem not from a lack of domain knowledge within the LLMs, but rather from the absence of an effective reasoning structure that guides the LLMs to elicit the right knowledge, incorporate the knowledge in step-by-step reasoning, and iteratively refine results for further improved quality. On this basis, we introduce StructChem, a simple yet effective prompting strategy that offers the desired guidance and substantially boosts the LLMs' chemical reasoning capability. Testing across four chemistry areas -- quantum chemistry, mechanics, physical chemistry, and kinetics -- StructChem substantially enhances GPT-4's performance, with up to 30\% peak improvement. Our analysis also underscores the unique difficulties of precise grounded reasoning in science with LLMs, highlighting a need for more research in this area. Code is available at \url{https://github.com/ozyyshr/StructChem}.

Figures (14)

• Figure 1: Proportions (%) of four error types ( #errors / #all-cases) for GPT-4 and StructChem. StructChem substantially reduces reasoning error.
• Figure 2: The illustration of (a) simple chemistry problem, (b) complex chemistry problem sampled from SciBench wang2023scibench, and (c) the zero-shot response from GPT-4 with chain-of-thought (CoT) DBLP:conf/nips/Wei0SBIXCLZ22 for the complex chemistry problem. The error types are illustrated corresponding to the definition in Figure \ref{['fig:intro-error']}: (I) irrelevant knowledge, (II) incorrect knowledge, (III) reasoning error, (IV) calculation error. We randomly select 100 error cases of GPT-4 (CoT) in SciBench.
• Figure 3: An in-depth illustration of the StructChem framework. When tackling a chemistry problem, we first utilize a structured instruction approach, resulting in "formulae generation" $\mathcal{F}_0$ and "step-by-step reasoning" $\mathcal{R}_0$. These generated segments are then fed to a thorough "confidence-based review-and-refinement" as initial input. The process is repeated $n$ times til getting reviewed formulae $\mathcal{F}_n$ and reasoning $\mathcal{R}_n$. Each iteration is guided by incorporating confidence scores $C_i$. $\rightarrow$ in "iterative review and refinement" denote the choice made for each iteration. Full instructions can be found in Figure \ref{['fig:prompt']}.
• Figure 4: Instruction for formulae generation (part of the structured instruction in Figure \ref{['fig:method']}) and the output for problem in Figure \ref{['fig:intro']}.
• Figure 5: Instruction for reasoning process (part of the structured instruction in Figure \ref{['fig:method']}) and the output for problem in Figure \ref{['fig:intro']}.