Crafting desirable climate trajectories with RL explored socio-environmental simulations

TL;DR

The findings show that cooperative agents in this framework can consistently chart pathways towards more desirable futures in terms of reduced carbon emissions and improved economy, however, upon introducing competition between agents, for instance by using opposing reward functions, desirable climate futures are rarely reached.

Abstract

Climate change poses an existential threat, necessitating effective climate policies to enact impactful change. Decisions in this domain are incredibly complex, involving conflicting entities and evidence. In the last decades, policymakers increasingly use simulations and computational methods to guide some of their decisions. Integrated Assessment Models (IAMs) are one of such methods, which combine social, economic, and environmental simulations to forecast potential policy effects. For example, the UN uses outputs of IAMs for their recent Intergovernmental Panel on Climate Change (IPCC) reports. Traditionally these have been solved using recursive equation solvers, but have several shortcomings, e.g. struggling at decision making under uncertainty. Recent preliminary work using Reinforcement Learning (RL) to replace the traditional solvers shows promising results in decision making in uncertain and noisy scenarios. We extend on this work by introducing multiple interacting RL agents as a preliminary analysis on modelling the complex interplay of socio-interactions between various stakeholders or nations that drives much of the current climate crisis. Our findings show that cooperative agents in this framework can consistently chart pathways towards more desirable futures in terms of reduced carbon emissions and improved economy. However, upon introducing competition between agents, for instance by using opposing reward functions, desirable climate futures are rarely reached. Modelling competition is key to increased realism in these simulations, as such we employ policy interpretation by visualising what states lead to more uncertain behaviour, to understand algorithm failure. Finally, we highlight the current limitations and avenues for further work to ensure future technology uptake for policy derivation.

Figures (13)

• Figure 1: The AYS model state space from kittel2021lakes. Translucent grey planes signify the two PBs, and the green and black points denote the fixed point end conditions for a single agent. Whisker lines indicate flow forces within the model, that tend towards either of the two fixed points. The colours showing the flow to the respective fixed points.
• Figure 2: Multi-agent AYS interaction cycle (diagram adapted from kittel2021lakes). Block arrows are positive interactions, dashed arrows are negative interactions.
• Figure 3: Homogeneous agent's win rates. Each experiment is run over six seeds with the line corresponding to mean win rate with translucent standard error bounds. Num agents relates to the number of agents in the simulation.
• Figure 4: Homogeneous agent's win rates for a longer range of training steps. These experiments are only run over two seeds due to computational constraints.
• Figure 5: Heterogeneous agent's win rates. We have omitted the single agent scenario as these results match between homogeneous and heterogeneous starting points. Each experiment is run over six seeds with the line corresponding to mean win rate with translucent standard error bounds.