Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PolyReal: A Benchmark for Real-World Polymer Science Workflows

Wanhao Liu, Weida Wang, Jiaqing Xie, Suorong Yang, Jue Wang, Benteng Chen, Guangtao Mei, Zonglin Yang, Shufei Zhang, Yuchun Mo, Lang Cheng, Jin Zeng, Houqiang Li, Wanli Ouyang, Yuqiang Li

Abstract

Multimodal Large Language Models (MLLMs) excel in general domains but struggle with complex, real-world science. We posit that polymer science, an interdisciplinary field spanning chemistry, physics, biology, and engineering, is an ideal high-stakes testbed due to its diverse multimodal data. Yet, existing benchmarks related to polymer science largely overlook real-world workflows, limiting their practical utility and failing to systematically evaluate MLLMs across the full, practice-grounded lifecycle of experimentation. We introduce PolyReal, a novel multimodal benchmark grounded in real-world scientific practices to evaluate MLLMs on the full lifecycle of polymer experimentation. It covers five critical capabilities: (1) foundational knowledge application; (2) lab safety analysis; (3) experiment mechanism reasoning; (4) raw data extraction; and (5) performance & application exploration. Our evaluation of leading MLLMs on PolyReal reveals a capability imbalance. While models perform well on knowledge-intensive reasoning (e.g., Experiment Mechanism Reasoning), they drop sharply on practice-based tasks (e.g., Lab Safety Analysis and Raw Data Extraction). This exposes a severe gap between abstract scientific knowledge and its practical, context-dependent application, showing that these real-world tasks remain challenging for MLLMs. Thus, PolyReal helps address this evaluation gap and provides a practical benchmark for assessing AI systems in real-world scientific workflows.

PolyReal: A Benchmark for Real-World Polymer Science Workflows

Abstract

Multimodal Large Language Models (MLLMs) excel in general domains but struggle with complex, real-world science. We posit that polymer science, an interdisciplinary field spanning chemistry, physics, biology, and engineering, is an ideal high-stakes testbed due to its diverse multimodal data. Yet, existing benchmarks related to polymer science largely overlook real-world workflows, limiting their practical utility and failing to systematically evaluate MLLMs across the full, practice-grounded lifecycle of experimentation. We introduce PolyReal, a novel multimodal benchmark grounded in real-world scientific practices to evaluate MLLMs on the full lifecycle of polymer experimentation. It covers five critical capabilities: (1) foundational knowledge application; (2) lab safety analysis; (3) experiment mechanism reasoning; (4) raw data extraction; and (5) performance & application exploration. Our evaluation of leading MLLMs on PolyReal reveals a capability imbalance. While models perform well on knowledge-intensive reasoning (e.g., Experiment Mechanism Reasoning), they drop sharply on practice-based tasks (e.g., Lab Safety Analysis and Raw Data Extraction). This exposes a severe gap between abstract scientific knowledge and its practical, context-dependent application, showing that these real-world tasks remain challenging for MLLMs. Thus, PolyReal helps address this evaluation gap and provides a practical benchmark for assessing AI systems in real-world scientific workflows.

Paper Structure

This paper contains 26 sections, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. PolyReal Dataset and Tasks
  4. Data Collection
  5. (1) Data Sourcing
  6. (2) Content Breadth
  7. (3) Question Format Curation
  8. (4) Expert Authoring and Validation
  9. Task
  10. Experiments and Evaluations
  11. Evaluation Setup
  12. Main Results
  13. Analysis
  14. Ablation
  15. Conclusion
  16. ...and 11 more sections

Figures (10)

  • Figure 1: Overview of the PolyReal benchmark and its five-stage workflow.
  • Figure 2: Overview of the PolyReal benchmark's composition, comprising 545 high-quality question-answer pairs: (a) Distribution of questions across the five core workflow modules covering the full research lifecycle; and (b) Comprehensive topic coverage, detailing the distribution across eight key sub-fields of polymer science.
  • Figure 3: Comparison of model performance across PolyReal, evaluated by F1 score.
  • Figure 4: Qualitative examples of MLLM failure modes: (Case 1) misapplying known principles and (Case 2) misinterpreting raw spectral data.
  • Figure 5: Comparison of model performance across the eight key polymer science sub-fields in PolyReal, evaluated by F1 score.
  • ...and 5 more figures