Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using Large Language Models for Solving Thermodynamic Problems

Rebecca Loubet, Pascal Zittlau, Luisa Vollmer, Marco Hoffmann, Sophie Fellenz, Fabian Jirasek, Heike Leitte, Hans Hasse

TL;DR

This study benchmarks five large language models on thermodynamics problem solving using two curated problem sets (simple and advanced), with three independent runs per problem and expert scoring. It reveals substantial variability in LLM outputs, with GPT-4 and GPT-4o performing best on simple tasks but all models faltering on advanced ones, highlighting limited analytical capabilities. The work contrasts LLM-based solving with KnowTD, a knowledge-based thermodynamics system, and discusses two hybrid integration strategies: using LLMs to enhance KnowTD inputs/outputs and leveraging KnowTD to tighten LLM reasoning through verification and prompting. The findings underscore the need for knowledge-guided, verifiable approaches to reliably tackle complex scientific and engineering problems with AI.

Abstract

Large Language Models (LLMs) have made significant progress in reasoning, demonstrating their capability to generate human-like responses. This study analyzes the problem-solving capabilities of LLMs in the domain of thermodynamics. A benchmark of 22 thermodynamic problems to evaluate LLMs is presented that contains both simple and advanced problems. Five different LLMs are assessed: GPT-3.5, GPT-4, and GPT-4o from OpenAI, Llama 3.1 from Meta, and le Chat from MistralAI. The answers of these LLMs were evaluated by trained human experts, following a methodology akin to the grading of academic exam responses. The scores and the consistency of the answers are discussed, together with the analytical skills of the LLMs. Both strengths and weaknesses of the LLMs become evident. They generally yield good results for the simple problems, but also limitations become clear: The LLMs do not provide consistent results, they often fail to fully comprehend the context and make wrong assumptions. Given the complexity and domain-specific nature of the problems, the statistical language modeling approach of the LLMs struggles with the accurate interpretation and the required reasoning. The present results highlight the need for more systematic integration of thermodynamic knowledge with LLMs, for example, by using knowledge-based methods.

Using Large Language Models for Solving Thermodynamic Problems

TL;DR

This study benchmarks five large language models on thermodynamics problem solving using two curated problem sets (simple and advanced), with three independent runs per problem and expert scoring. It reveals substantial variability in LLM outputs, with GPT-4 and GPT-4o performing best on simple tasks but all models faltering on advanced ones, highlighting limited analytical capabilities. The work contrasts LLM-based solving with KnowTD, a knowledge-based thermodynamics system, and discusses two hybrid integration strategies: using LLMs to enhance KnowTD inputs/outputs and leveraging KnowTD to tighten LLM reasoning through verification and prompting. The findings underscore the need for knowledge-guided, verifiable approaches to reliably tackle complex scientific and engineering problems with AI.

Abstract

Large Language Models (LLMs) have made significant progress in reasoning, demonstrating their capability to generate human-like responses. This study analyzes the problem-solving capabilities of LLMs in the domain of thermodynamics. A benchmark of 22 thermodynamic problems to evaluate LLMs is presented that contains both simple and advanced problems. Five different LLMs are assessed: GPT-3.5, GPT-4, and GPT-4o from OpenAI, Llama 3.1 from Meta, and le Chat from MistralAI. The answers of these LLMs were evaluated by trained human experts, following a methodology akin to the grading of academic exam responses. The scores and the consistency of the answers are discussed, together with the analytical skills of the LLMs. Both strengths and weaknesses of the LLMs become evident. They generally yield good results for the simple problems, but also limitations become clear: The LLMs do not provide consistent results, they often fail to fully comprehend the context and make wrong assumptions. Given the complexity and domain-specific nature of the problems, the statistical language modeling approach of the LLMs struggles with the accurate interpretation and the required reasoning. The present results highlight the need for more systematic integration of thermodynamic knowledge with LLMs, for example, by using knowledge-based methods.

Paper Structure

This paper contains 15 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Studied LLMs and their Utilization
  3. Benchmark Problems and Evaluation
  4. Overview
  5. Simple Problems
  6. Advanced Problems
  7. Evaluation
  8. Results and Discussion
  9. Performance Evaluation
  10. Error Types
  11. Preliminary tests with OpenAI's o1-preview model
  12. Knowledge Based Systems for Thermodynamic Problem Solving
  13. Using LLMs to enhance the performance of KnowTD
  14. Using KnowTD to enhance the performance of LLMs
  15. Conclusion

Figures (2)

  • Figure 1: Scores of the different LLMs for the 13 simple problems. Blue bars indicate the maximum number of points, red bars indicate the scores obtained in the three independent runs.
  • Figure 2: Scores of the different LLMs for the nine advanced problems. Blue bars indicate the maximum number of points, red bars indicate the scores obtained in the three independent runs.