Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Analysis of Sustainability Reports: Using Large Language Models for the Extraction and Prediction of EU Taxonomy-Compliant KPIs

Jonathan Schmoll, Adam Jatowt

TL;DR

This paper tackles the challenge of EU Taxonomy compliance reporting by creating a public benchmark dataset and evaluating large language models (LLMs) on a four-task information extraction pipeline. It introduces a structured dataset derived from 190 company reports with ground truth activities and KPIs and formulates four tasks: multiclass activity prediction, binary activity classification, KPI regression, and a multi-step agentic workflow. The results show a clear gap between qualitative and quantitative tasks: LLMs achieve only moderate success in identifying activities and fail at zero-shot KPI prediction, with a paradox of context where concise metadata often outperforms full reports and confidence scores are poorly calibrated. The study concludes that LLMs are best used as assistive tools within human-in-the-loop processes, and it provides a public benchmark to spur development of domain-tuned models for ESG and regulatory analysis.

Abstract

The manual, resource-intensive process of complying with the EU Taxonomy presents a significant challenge for companies. While Large Language Models (LLMs) offer a path to automation, research is hindered by a lack of public benchmark datasets. To address this gap, we introduce a novel, structured dataset from 190 corporate reports, containing ground-truth economic activities and quantitative Key Performance Indicators (KPIs). We use this dataset to conduct the first systematic evaluation of LLMs on the core compliance workflow. Our results reveal a clear performance gap between qualitative and quantitative tasks. LLMs show moderate success in the qualitative task of identifying economic activities, with a multi-step agentic framework modestly enhancing precision. Conversely, the models comprehensively fail at the quantitative task of predicting financial KPIs in a zero-shot setting. We also discover a paradox, where concise metadata often yields superior performance to full, unstructured reports, and find that model confidence scores are poorly calibrated. We conclude that while LLMs are not ready for full automation, they can serve as powerful assistive tools for human experts. Our dataset provides a public benchmark for future research.

Automated Analysis of Sustainability Reports: Using Large Language Models for the Extraction and Prediction of EU Taxonomy-Compliant KPIs

TL;DR

This paper tackles the challenge of EU Taxonomy compliance reporting by creating a public benchmark dataset and evaluating large language models (LLMs) on a four-task information extraction pipeline. It introduces a structured dataset derived from 190 company reports with ground truth activities and KPIs and formulates four tasks: multiclass activity prediction, binary activity classification, KPI regression, and a multi-step agentic workflow. The results show a clear gap between qualitative and quantitative tasks: LLMs achieve only moderate success in identifying activities and fail at zero-shot KPI prediction, with a paradox of context where concise metadata often outperforms full reports and confidence scores are poorly calibrated. The study concludes that LLMs are best used as assistive tools within human-in-the-loop processes, and it provides a public benchmark to spur development of domain-tuned models for ESG and regulatory analysis.

Abstract

The manual, resource-intensive process of complying with the EU Taxonomy presents a significant challenge for companies. While Large Language Models (LLMs) offer a path to automation, research is hindered by a lack of public benchmark datasets. To address this gap, we introduce a novel, structured dataset from 190 corporate reports, containing ground-truth economic activities and quantitative Key Performance Indicators (KPIs). We use this dataset to conduct the first systematic evaluation of LLMs on the core compliance workflow. Our results reveal a clear performance gap between qualitative and quantitative tasks. LLMs show moderate success in the qualitative task of identifying economic activities, with a multi-step agentic framework modestly enhancing precision. Conversely, the models comprehensively fail at the quantitative task of predicting financial KPIs in a zero-shot setting. We also discover a paradox, where concise metadata often yields superior performance to full, unstructured reports, and find that model confidence scores are poorly calibrated. We conclude that while LLMs are not ready for full automation, they can serve as powerful assistive tools for human experts. Our dataset provides a public benchmark for future research.
Paper Structure (39 sections, 12 figures, 9 tables)

This paper contains 39 sections, 12 figures, 9 tables.

Table of Contents

  1. Introduction
  2. The EU Taxonomy Reporting Task
  3. Regulatory Context and Reporting Obligations
  4. Formulating the IE Tasks: Activity Classification and KPI Regression
  5. Related Work
  6. NLP for Financial and ESG Reporting
  7. Information Extraction from Regulatory Documents
  8. A Benchmark Dataset for EU Taxonomy Analysis
  9. Dataset Curation and Preprocessing
  10. Dataset Structure and Statistics
  11. Dataset Composition
  12. Reporting Statistics
  13. Methodology: Evaluating LLMs for Taxonomy Reporting
  14. Models and Experimental Setup
  15. Task 1: Multiclass Activity Prediction
  16. ...and 24 more sections

Figures (12)

  • Figure 1: Overview: Extraction of source data from public reports. Multiple zero-shot experiments against the dataset to test current model capabilities.
  • Figure 2: Architectural diagrams of the two agentic workflows. Approach 1 (left) is a sequential pipeline. Approach 2 (right) uses a parallel structure to refine predictions.
  • Figure 3: Comparison of ground-truth (orange) vs. predicted (green) KPI distributions. The model (gemini-flash-2.5) fails to capture the true data distributions.
  • Figure 4: Detailed distributions of the six EU Taxonomy KPI percentages. The histograms for aligned KPIs (right column) show a strong concentration at or near zero, while eligible KPIs (left column) exhibit more varied distributions.
  • Figure 5: Pair plot matrix of the six EU Taxonomy KPI percentages. The diagonal shows the kernel density estimate (KDE) for each variable, while the off-diagonal plots show the scatter relationship between each pair of variables.
  • ...and 7 more figures