Optimal CO2 storage management considering safety constraints in multi-stakeholder multi-site CCS projects: a Markov game perspective

TL;DR

This paper models basin-scale geological carbon storage (GCS) as a constrained Markov game (CMG) to capture inter-operator dependencies across multi-site projects and safety constraints. A safe multi-agent reinforcement learning (safe MARL) approach, specifically MADDPG with centralized training and decentralized execution, is combined with an Embed-to-Control (E2C) surrogate to efficiently learn policies under pore-pressure safety limits. The study evaluates five coalition structures (including fully cooperative and various mixed coalitions) and demonstrates that grand coalition policies maximize total NPV ($12{,}191$ M) compared with fully competitive outcomes ($8{,}942$ M), while still respecting safety constraints; mixed coalitions show asymmetric payoff distributions favoring coalition members. When compared to constrained multi-objective optimization (CMOO) via NSGA-II, the fully cooperative MARL solution is non-dominated within the discrete set and achieves the highest system-wide efficiency, highlighting the value of coordination in basin-scale CCS operations. The framework offers a scalable, safety-aware decision-making tool for basin-scale CCS planning and provides insight into how collaboration structures influence economic outcomes and risk containment in multi-stakeholder energy transition projects.

Abstract

Carbon capture and storage (CCS) projects typically involve a diverse array of stakeholders or players from public, private, and regulatory sectors, each with different objectives and responsibilities. Given the complexity, scale, and long-term nature of CCS operations, determining whether individual stakeholders can independently maximize their interests or whether collaborative coalition agreements are needed remains a central question for effective CCS project planning and management. CCS projects are often implemented in geologically connected sites, where shared geological features such as pressure space and reservoir pore capacity can lead to competitive behavior among stakeholders. Furthermore, CO2 storage sites are often located in geologically mature basins that previously served as sites for hydrocarbon extraction or wastewater disposal in order to leverage existing infrastructures, which makes unilateral optimization even more complicated and unrealistic. In this work, we propose a paradigm based on Markov games to quantitatively investigate how different coalition structures affect the goals of stakeholders. We frame this multi-stakeholder multi-site problem as a multi-agent reinforcement learning problem with safety constraints. Our approach enables agents to learn optimal strategies while compliant with safety regulations. We present an example where multiple operators are injecting CO2 into their respective project areas in a geologically connected basin. To address the high computational cost of repeated simulations of high-fidelity models, a previously developed surrogate model based on the Embed-to-Control (E2C) framework is employed. Our results demonstrate the effectiveness of the proposed framework in addressing optimal management of CO2 storage when multiple stakeholders with various objectives and goals are involved.

Figures (15)

• Figure 1: Relations between various stakeholders in a CCS project
• Figure 2: Sketch plot of MOO with two decision variable and two objectives. The gray areas represents the feasible decision and solution space. The blue dots in the objective space are Pareto optimal solutions forming the Pareto front, where improving one objective (an agent’s reward) necessarily degrades the other. Adapted from pereira2022review
• Figure 3: Schematic diagrams of (a) Markov decision process and (b) Markov game, which correspond to the frameworks for (a) single- and (b) multi-agent RL, respectively. Adapted from zhang2021multi
• Figure 4: Schematic diagrams of constrained Markov game, which correspond to multi-agent RL with safety constraints. Note that for every agent, there can be $m$ constraints, for simplicity, we assume there is only one constraint when $m=1$.
• Figure 5: A safe multi-agent reinforcement learning framework tailored for multi-stakeholder CCS projects. Each agent/operator is interacting with subsurface dynamical environment, receiving immediate reward (defined by present values) and penalty (e.g. injection pressure exceeding the fracturing pressure) and driving the environment to next state.