Machine Learning for Climate Policy: Understanding Policy Progression in the European Green Deal

TL;DR

The study addresses predicting the progression of EU climate policies within the European Green Deal by constructing a 165-policy dataset and comparing text representations (TF-IDF, BERT, ClimateBERT) with metadata features. It treats policy progression as an ordinal regression task mapped to a $0-1$ scale and evaluates multiple regression models, finding that BERT with Bayesian Ridge performs best when using both text and metadata ($RMSE$ = 0.16, $R^2$ = 0.38), while ClimateBERT excels on text alone ($RMSE$ = 0.17, $R^2$ = 0.29). Explainability analyses (permutation importance and SHAP) reveal that metadata, especially political party representation, substantially influences predictions, with domain-specific terms like environment and commission also contributing. The work demonstrates the potential of ML tools to support climate policy analysis and decision-making, while also highlighting limitations such as small sample size, class imbalance, and the need for careful interpretation alongside expert input.

Abstract

Climate change demands effective legislative action to mitigate its impacts. This study explores the application of machine learning (ML) to understand the progression of climate policy from announcement to adoption, focusing on policies within the European Green Deal. We present a dataset of 165 policies, incorporating text and metadata. We aim to predict a policy's progression status, and compare text representation methods, including TF-IDF, BERT, and ClimateBERT. Metadata features are included to evaluate the impact on predictive performance. On text features alone, ClimateBERT outperforms other approaches (RMSE = 0.17, R^2 = 0.29), while BERT achieves superior performance with the addition of metadata features (RMSE = 0.16, R^2 = 0.38). Using methods from explainable AI highlights the influence of factors such as policy wording and metadata including political party and country representation. These findings underscore the potential of ML tools in supporting climate policy analysis and decision-making.

Figures (5)

• Figure 1: Processing pipeline, from policy and metadata data collection to category prediction and associated explanations.
• Figure 2: Categories of policy stages considered, in the path towards adoption.
• Figure 3: Distribution of policy stage labels in the dataset.
• Figure 4: Feature permutation of BERT with Bayesian Ridge regression. Metadata features are colour-coded and text features are presented in black. Values are shown as the negative change in RMSE after shuffling each feature, so larger values indicate greater importance.
• Figure 5: SHapley Additive exPlanation (SHAP) values of TF-IDF with CatBoost regression (RMSE = 0.18, R$^2$ = 0.2). Metadata features are colour-coded and text features are presented in black. Feature values are colour-coded from blue to red. For text features, high TF-IDF values indicate low frequency of words in policy texts. Positive SHAP values indicate feature contribution towards advanced policy progression categories during training.