Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Empowering Consumers through Sentence-level Readability Scoring in German ESG Reports

Benjamin Josef Schüßler, Jakob Prange

Abstract

With the ever-growing urgency of sustainability in the economy and society, and the massive stream of information that comes with it, consumers need reliable access to that information. To address this need, companies began publishing so called Environmental, Social, and Governance (ESG) reports, both voluntarily and forced by law. To serve the public, these reports must be addressed not only to financial experts but also to non-expert audiences. But are they written clearly enough? In this work, we extend an existing sentence-level dataset of German ESG reports with crowdsourced readability annotations. We find that, in general, native speakers perceive sentences in ESG reports as easy to read, but also that readability is subjective. We apply various readability scoring methods and evaluate them regarding their prediction error and correlation with human rankings. Our analysis shows that, while LLM prompting has potential for distinguishing clear from hard-to-read sentences, a small finetuned transformer predicts human readability with the lowest error. Averaging predictions of multiple models can slightly improve the performance at the cost of slower inference.

Towards Empowering Consumers through Sentence-level Readability Scoring in German ESG Reports

Abstract

With the ever-growing urgency of sustainability in the economy and society, and the massive stream of information that comes with it, consumers need reliable access to that information. To address this need, companies began publishing so called Environmental, Social, and Governance (ESG) reports, both voluntarily and forced by law. To serve the public, these reports must be addressed not only to financial experts but also to non-expert audiences. But are they written clearly enough? In this work, we extend an existing sentence-level dataset of German ESG reports with crowdsourced readability annotations. We find that, in general, native speakers perceive sentences in ESG reports as easy to read, but also that readability is subjective. We apply various readability scoring methods and evaluate them regarding their prediction error and correlation with human rankings. Our analysis shows that, while LLM prompting has potential for distinguishing clear from hard-to-read sentences, a small finetuned transformer predicts human readability with the lowest error. Averaging predictions of multiple models can slightly improve the performance at the cost of slower inference.

Paper Structure

This paper contains 60 sections, 5 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Readability of German texts.
  4. Readability of ESG reports.
  5. Methodologically
  6. Data
  7. Readability Annotation.
  8. Agreement.
  9. Score distribution.
  10. Whitebox Readability Models
  11. Syntactic Features
  12. Part-of-Speech-tag $n$-grams.
  13. Depth of the dependency tree.
  14. Mean dependency distance.
  15. Part-of-Speech-tag of the root.
  16. ...and 45 more sections

Figures (5)

  • Figure 1: Readability as a foundation of consumer empowerment from ESG reports.
  • Figure 2: Structure of our syntax-based ARA model.
  • Figure 3: Two examples of different length and superficial complexity (translated from German).
  • Figure 4: Time-performance trade-offs. All axes rank models top-down from best to worst.
  • Figure 5: Prompt for the LLM-based ARA model using single-shot prompting.