An Expert-grounded benchmark of General Purpose LLMs in LCA

TL;DR

The paper presents the first expert-grounded benchmark of general-purpose LLMs for life cycle assessment (LCA), addressing the lack of ground truth and standardized evaluation in the field. It assesses eleven LLMs across 22 LCA tasks using 168 expert reviews, evaluating scientific accuracy, explainability, robustness, verifiability, and instruction adherence. Key findings reveal a substantial rate of inaccuracies (about 37%), notable hallucination in citations (up to 40% in some models), and no clear advantage for open-weight versus closed-weight models, though some open-weight models perform competitively. The work highlights significant risks in naive LLM use for LCA, while also demonstrating potential benefits in explanation quality and efficiency, and it outlines concrete directions—such as grounding mechanisms, retrieval-augmented generation, and larger, open benchmarks—to advance safe and trustworthy AI-assisted LCA practice.

Abstract

Purpose: Artificial intelligence (AI), and in particular large language models (LLMs), are increasingly being explored as tools to support life cycle assessment (LCA). While demonstrations exist across environmental and social domains, systematic evidence on their reliability, robustness, and usability remains limited. This study provides the first expert-grounded benchmark of LLMs in LCA, addressing the absence of standardized evaluation frameworks in a field where no clear ground truth or consensus protocols exist. Methods: We evaluated eleven general-purpose LLMs, spanning both commercial and open-source families, across 22 LCA-related tasks. Seventeen experienced practitioners reviewed model outputs against criteria directly relevant to LCA practice, including scientific accuracy, explanation quality, robustness, verifiability, and adherence to instructions. We collected 168 expert reviews. Results: Experts judged 37% of responses to contain inaccurate or misleading information. Ratings of accuracy and quality of explanation were generally rated average or good on many models even smaller models, and format adherence was generally rated favourably. Hallucination rates varied significantly, with some models producing hallucinated citations at rates of up to 40%. There was no clear-cut distinction between ratings on open-weight versus closed-weight LLMs, with open-weight models outperforming or competing on par with closed-weight models on criteria such as accuracy and quality of explanation. Conclusion: These findings highlight the risks of applying LLMs naïvely in LCA, such as when LLMs are treated as free-form oracles, while also showing benefits especially around quality of explanation and alleviating labour intensiveness of simple tasks. The use of general-purpose LLMs without grounding mechanisms presents ...

Figures (11)

• Figure 1: Methodology followed in conducting the benchmark creation and evaluation
• Figure 2: A heatmap where colours show the mean ratings by human experts on a range of criteria, where 1 (red) is worst and 4 (green) is best. Scores for correctness and format adherence are scaled from the interval [0,1] to [1,4]. Correctness results are the inverted results from the binary question on inaccuracy (sec. \ref{['correctness_calc']}). The right-most column shows the number of expert reviews included. Asterisks next to model names indicates open weight LLMs.
• Figure 3: Bar chart showing mean accuracy (green, hatched) and mean explanation (red, circled) scores given by expert reviewers. Ordered from left to right in order of decreasing average score across accuracy and explanation. Error bars indicate standard error. Asterisks indicate open-weight models.
• Figure 4: Results from question 2 – "How scientifically accurate is this response - on a scale from 1-4?". n signifies the number of individual human-reviewed responses per model, from a panel of LCA experts. Asterisks denote open-weight models. Error bars indicate standard error. Asterisks indicate open-weight models.
• Figure 5: Results from question 3 – "How well explained is the answer - on a scale from 1-4?”. n signifies the number of individual human-reviewed responses per model, from a panel of LCA experts. Error bars indicate standard error. Asterisks indicate open-weight models.