Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bridging AI and Carbon Capture: A Dataset for LLMs in Ionic Liquids and CBE Research

Gaurab Sarkar, Sougata Saha

TL;DR

The paper addresses the challenge of assessing large language models in specialized Chemical and Biological Engineering, focusing on Ionic Liquids for CO2 capture. It introduces a 5,920-example textual entailment benchmark derived from 74 expert claims to probe both factual knowledge and domain-specific reasoning in open-weight LLMs under 10B parameters. The results show that while models like Llama possess substantive IL and carbon capture knowledge, their domain-specific reasoning remains limited, particularly under linguistic perturbations and adversarial options. The authors propose domain-aligned training strategies such as fine-tuning, PEFT with LoRA, and retrieval-augmented approaches, while highlighting the environmental and societal stakes of aligning AI development with climate goals toward carbon neutrality by 2050.

Abstract

Large Language Models (LLMs) have demonstrated exceptional performance in general knowledge and reasoning tasks across various domains. However, their effectiveness in specialized scientific fields like Chemical and Biological Engineering (CBE) remains underexplored. Addressing this gap requires robust evaluation benchmarks that assess both knowledge and reasoning capabilities in these niche areas, which are currently lacking. To bridge this divide, we present a comprehensive empirical analysis of LLM reasoning capabilities in CBE, with a focus on Ionic Liquids (ILs) for carbon sequestration - an emerging solution for mitigating global warming. We develop and release an expert - curated dataset of 5,920 examples designed to benchmark LLMs' reasoning in this domain. The dataset incorporates varying levels of difficulty, balancing linguistic complexity and domain-specific knowledge. Using this dataset, we evaluate three open-source LLMs with fewer than 10 billion parameters. Our findings reveal that while smaller general-purpose LLMs exhibit basic knowledge of ILs, they lack the specialized reasoning skills necessary for advanced applications. Building on these results, we discuss strategies to enhance the utility of LLMs for carbon capture research, particularly using ILs. Given the significant carbon footprint of LLMs, aligning their development with IL research presents a unique opportunity to foster mutual progress in both fields and advance global efforts toward achieving carbon neutrality by 2050.

Bridging AI and Carbon Capture: A Dataset for LLMs in Ionic Liquids and CBE Research

TL;DR

The paper addresses the challenge of assessing large language models in specialized Chemical and Biological Engineering, focusing on Ionic Liquids for CO2 capture. It introduces a 5,920-example textual entailment benchmark derived from 74 expert claims to probe both factual knowledge and domain-specific reasoning in open-weight LLMs under 10B parameters. The results show that while models like Llama possess substantive IL and carbon capture knowledge, their domain-specific reasoning remains limited, particularly under linguistic perturbations and adversarial options. The authors propose domain-aligned training strategies such as fine-tuning, PEFT with LoRA, and retrieval-augmented approaches, while highlighting the environmental and societal stakes of aligning AI development with climate goals toward carbon neutrality by 2050.

Abstract

Large Language Models (LLMs) have demonstrated exceptional performance in general knowledge and reasoning tasks across various domains. However, their effectiveness in specialized scientific fields like Chemical and Biological Engineering (CBE) remains underexplored. Addressing this gap requires robust evaluation benchmarks that assess both knowledge and reasoning capabilities in these niche areas, which are currently lacking. To bridge this divide, we present a comprehensive empirical analysis of LLM reasoning capabilities in CBE, with a focus on Ionic Liquids (ILs) for carbon sequestration - an emerging solution for mitigating global warming. We develop and release an expert - curated dataset of 5,920 examples designed to benchmark LLMs' reasoning in this domain. The dataset incorporates varying levels of difficulty, balancing linguistic complexity and domain-specific knowledge. Using this dataset, we evaluate three open-source LLMs with fewer than 10 billion parameters. Our findings reveal that while smaller general-purpose LLMs exhibit basic knowledge of ILs, they lack the specialized reasoning skills necessary for advanced applications. Building on these results, we discuss strategies to enhance the utility of LLMs for carbon capture research, particularly using ILs. Given the significant carbon footprint of LLMs, aligning their development with IL research presents a unique opportunity to foster mutual progress in both fields and advance global efforts toward achieving carbon neutrality by 2050.
Paper Structure (13 sections, 7 figures, 1 table)

This paper contains 13 sections, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Ionic Liquids for Carbon Capture
  4. LLMs for Scientific Research
  5. A Practical Test for Knowledge
  6. Argument Structures
  7. The Entailment Task
  8. Dataset Creation
  9. Experiments
  10. Results and Observations
  11. Discussion
  12. Conclusion
  13. Appendix

Figures (7)

  • Figure 1: Dataset Creation Pipeline
  • Figure 2: Model-wise precision and recall for experiments in Group 2 (left) and Group 3 (right).
  • Figure 3: Model-wise precision and recall for experiments in Group 4 (left) and Group 5 (right).
  • Figure 4: Model-wise precision, recall, and F1 for experiments in comparison suite 5.
  • Figure 5: Mistral 3-shot prompt to automatically extract and generate the missing assumptions from claims.
  • ...and 2 more figures