Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FGBench: A Dataset and Benchmark for Molecular Property Reasoning at Functional Group-Level in Large Language Models

Xuan Liu, Siru Ouyang, Xianrui Zhong, Jiawei Han, Huimin Zhao

TL;DR

FGBench introduces a large-scale FG-level dataset (625K QA pairs) and a validation-by-reconstruction pipeline to study molecular property reasoning in LLMs. It formalizes FG-centered tasks across single FG impacts, FG interactions, and molecular comparisons, and provides a 7K-subset benchmark across diverse models to reveal current reasoning gaps. The work demonstrates that state-of-the-art LLMs struggle with FG-level reasoning and FG interactions, highlighting the need for structure-aware training and multimodal approaches. This dataset and framework offer a foundation for developing interpretable, FG-informed chemistry LLMs with potential benefits for SAR analyses and drug discovery.

Abstract

Large language models (LLMs) have gained significant attention in chemistry. However, most existing datasets center on molecular-level property prediction and overlook the role of fine-grained functional group (FG) information. Incorporating FG-level data can provide valuable prior knowledge that links molecular structures with textual descriptions, which can be used to build more interpretable, structure-aware LLMs for reasoning on molecule-related tasks. Moreover, LLMs can learn from such fine-grained information to uncover hidden relationships between specific functional groups and molecular properties, thereby advancing molecular design and drug discovery. Here, we introduce FGBench, a dataset comprising 625K molecular property reasoning problems with functional group information. Functional groups are precisely annotated and localized within the molecule, which ensures the dataset's interoperability thereby facilitating further multimodal applications. FGBench includes both regression and classification tasks on 245 different functional groups across three categories for molecular property reasoning: (1) single functional group impacts, (2) multiple functional group interactions, and (3) direct molecular comparisons. In the benchmark of state-of-the-art LLMs on 7K curated data, the results indicate that current LLMs struggle with FG-level property reasoning, highlighting the need to enhance reasoning capabilities in LLMs for chemistry tasks. We anticipate that the methodology employed in FGBench to construct datasets with functional group-level information will serve as a foundational framework for generating new question-answer pairs, enabling LLMs to better understand fine-grained molecular structure-property relationships. The dataset and evaluation code are available at https://github.com/xuanliugit/FGBench.

FGBench: A Dataset and Benchmark for Molecular Property Reasoning at Functional Group-Level in Large Language Models

TL;DR

FGBench introduces a large-scale FG-level dataset (625K QA pairs) and a validation-by-reconstruction pipeline to study molecular property reasoning in LLMs. It formalizes FG-centered tasks across single FG impacts, FG interactions, and molecular comparisons, and provides a 7K-subset benchmark across diverse models to reveal current reasoning gaps. The work demonstrates that state-of-the-art LLMs struggle with FG-level reasoning and FG interactions, highlighting the need for structure-aware training and multimodal approaches. This dataset and framework offer a foundation for developing interpretable, FG-informed chemistry LLMs with potential benefits for SAR analyses and drug discovery.

Abstract

Large language models (LLMs) have gained significant attention in chemistry. However, most existing datasets center on molecular-level property prediction and overlook the role of fine-grained functional group (FG) information. Incorporating FG-level data can provide valuable prior knowledge that links molecular structures with textual descriptions, which can be used to build more interpretable, structure-aware LLMs for reasoning on molecule-related tasks. Moreover, LLMs can learn from such fine-grained information to uncover hidden relationships between specific functional groups and molecular properties, thereby advancing molecular design and drug discovery. Here, we introduce FGBench, a dataset comprising 625K molecular property reasoning problems with functional group information. Functional groups are precisely annotated and localized within the molecule, which ensures the dataset's interoperability thereby facilitating further multimodal applications. FGBench includes both regression and classification tasks on 245 different functional groups across three categories for molecular property reasoning: (1) single functional group impacts, (2) multiple functional group interactions, and (3) direct molecular comparisons. In the benchmark of state-of-the-art LLMs on 7K curated data, the results indicate that current LLMs struggle with FG-level property reasoning, highlighting the need to enhance reasoning capabilities in LLMs for chemistry tasks. We anticipate that the methodology employed in FGBench to construct datasets with functional group-level information will serve as a foundational framework for generating new question-answer pairs, enabling LLMs to better understand fine-grained molecular structure-property relationships. The dataset and evaluation code are available at https://github.com/xuanliugit/FGBench.

Paper Structure

This paper contains 36 sections, 4 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. FGBench: Task and Construction
  4. Problem Definition
  5. Dataset Construction
  6. Reasoning Task Categories and QA Pairs
  7. Benchmark Large Language Models on FGBench
  8. Experimental Setup
  9. Main Result
  10. Failure Analysis
  11. Conclusion and Outlook
  12. Limitation
  13. FGBench construction details
  14. The definition of functional groups
  15. Complete dataset and tasks used in FGBench
  16. ...and 21 more sections

Figures (4)

  • Figure 1: Illustration of LLMs for two molecular property prediction tasks, (a) molecular property prediction and (b) molecular property reasoning.
  • Figure 2: FGBench dataset construction workflow. Molecules from a database are canonicalized and compared to generate a similarity matrix. Molecule pairs exceeding a similarity threshold are selected, and their FG differences are calculated. These differences are validated through a reconstruction process that involves removing and replacing FGs, ensuring structural consistency. The final output is a curated molecule comparison dataset.
  • Figure 3: An example of multiple FG interactions from the BACE database. The ether group is deleted from the initial molecule, and an amine group is attached to it. The change related to these two functional groups causes the change of the molecule's BACE-1 inhibitory activity from active to inactive.
  • Figure 4: An example output from o3-mini on a single FG impact QA (Boolean) from Lipophilicity dataset. Left is the original molecule. Right is the molecule after adding a nitrile (-CN) group. The o3-mini response is in the gray box.