Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Teaching and Evaluating LLMs to Reason About Polymer Design Related Tasks

Dikshya Mohanty, Mohammad Saqib Hasan, Syed Mostofa Monsur, Size Zheng, Benjamin Hsiao, Niranjan Balasubramanian

TL;DR

This work introduces PolyBench, a large-scale, ground-truth benchmark (over $125k$ polymer-design tasks drawn from $13$M+ data points) to study LLM-based reasoning in polymer design. It combines a knowledge-augmented distillation pipeline with six task categories to teach models structural understanding, property inference, and synthesis planning, while providing explicit chain-of-thought traces grounded in rich polymer data. Experiments show that small LLMs finetuned on PolyBench outperform comparable baselines and even some frontier models on the PolyBench test set, and generalize to external polymer benchmarks. The results reveal a compositionality gap in current models and demonstrate that targeted domain alignment and reasoning traces significantly improve polymer-design capabilities, with substantial implications for AI-assisted polymer discovery and design.

Abstract

Research in AI4Science has shown promise in many science applications, including polymer design. However, current LLMs prove ineffective on this problem space because: (i) most models lack polymer-specific knowledge (ii) existing aligned models lack coverage of knowledge and capabilities relevant to polymer design. Addressing this, we introduce PolyBench, a large scale training and test benchmark dataset of more than 125K polymer design related tasks, leveraging a knowledge base of 13M+ data points obtained from experimental and synthetic sources to ensure broad coverage of polymers and their properties. For effective alignment using PolyBench, we introduce a knowledge-augmented reasoning distillation method that augments this dataset with structured CoT. Furthermore, tasks in PolyBench are organized from simple to complex analytical reasoning problems, enabling generalization tests and diagnostic probes across the problem space. Experiments show that small language models (SLMs), of 7B to 14B parameters, trained on PolyBench data outperform similar sized models, and even closed source frontier LLMs on PolyBench test dataset while demonstrating gains on other polymer benchmarks as well.

Teaching and Evaluating LLMs to Reason About Polymer Design Related Tasks

TL;DR

This work introduces PolyBench, a large-scale, ground-truth benchmark (over polymer-design tasks drawn from M+ data points) to study LLM-based reasoning in polymer design. It combines a knowledge-augmented distillation pipeline with six task categories to teach models structural understanding, property inference, and synthesis planning, while providing explicit chain-of-thought traces grounded in rich polymer data. Experiments show that small LLMs finetuned on PolyBench outperform comparable baselines and even some frontier models on the PolyBench test set, and generalize to external polymer benchmarks. The results reveal a compositionality gap in current models and demonstrate that targeted domain alignment and reasoning traces significantly improve polymer-design capabilities, with substantial implications for AI-assisted polymer discovery and design.

Abstract

Research in AI4Science has shown promise in many science applications, including polymer design. However, current LLMs prove ineffective on this problem space because: (i) most models lack polymer-specific knowledge (ii) existing aligned models lack coverage of knowledge and capabilities relevant to polymer design. Addressing this, we introduce PolyBench, a large scale training and test benchmark dataset of more than 125K polymer design related tasks, leveraging a knowledge base of 13M+ data points obtained from experimental and synthetic sources to ensure broad coverage of polymers and their properties. For effective alignment using PolyBench, we introduce a knowledge-augmented reasoning distillation method that augments this dataset with structured CoT. Furthermore, tasks in PolyBench are organized from simple to complex analytical reasoning problems, enabling generalization tests and diagnostic probes across the problem space. Experiments show that small language models (SLMs), of 7B to 14B parameters, trained on PolyBench data outperform similar sized models, and even closed source frontier LLMs on PolyBench test dataset while demonstrating gains on other polymer benchmarks as well.
Paper Structure (88 sections, 3 equations, 4 figures, 13 tables)

This paper contains 88 sections, 3 equations, 4 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Design Related Tasks Grounded in Scientific Data
  3. Task Design and Organization
  4. Knowledge-Augmented Reasoning and Distillation
  5. Benchmarking and Understanding gaps in model reasoning
  6. Related Work
  7. Benchmarks for Chemistry Reasoning
  8. LLMs for Chemistry
  9. LLMs for Polymer Tasks
  10. LLMs for Polymer Design
  11. Challenges
  12. PolyBench
  13. Polymer Tasks Design and Organization
  14. Task Generation
  15. Knowledge-Augmented Distillation
  16. ...and 73 more sections

Figures (4)

  • Figure 1: Example from PolyBench with sub-tasks. The central prompt requires jointly satisfying multiple constraints (highlighted in green), spanning reaction/synthesis feasibility, property targets, and functional/structural constraints. SubQuestions (SubQ) shows with associated skill tags (e.g., structure, rules, inference) to illustrate both the multi-constraint nature of polymer design and the compositionality/skill-gap diagnostics used in our analysis.
  • Figure 2: PolyData Creation Pipeline. We aggregate data from open-source databases covering $13$m polymers, standardize notations and properties based on SMEs' inputs, and augment with RDKit-computed features. The data is split into train/dev/test sets with non-overlapping polymers to ensure out-of-distribution evaluation. Each instance undergoes chain-of-thought distillation where polymer profiles are provided to teacher models via knowledge-augmented prompting. Generated reasoning traces are verified through automated judgment and manual review, producing (instruction,question,CoT,solution) tuples across six task categories.
  • Figure 3: CoT quality ratings of our KD Pipeline against GPT-4o and Sonnet for Accuracy, Completeness, and Relevance, on a 1-5 Likert scale
  • Figure 4: Human preference study on model outputs. Blind annotators select the preferred response among Phi-4-14B+CoT (PolyLM), GPT-4o, and Claude-3.5-Sonnet on a held-out set of PolyBench questions. Preference rates indicate that Phi-4-14B+CoT is chosen more often overall, supporting the automated evaluation trends.