Game Theory and Multi-Agent Reinforcement Learning for Zonal Ancillary Markets

TL;DR

Problem addressed: zonal ancillary markets with inter-zone coupling are modeled as a Stackelberg game that induces a nonconvex generalized Nash game with side constraints, Γ. Main approach: establish existence under mild conditions and exploit a generalized potential structure $P(w)=\sum_{n=1}^N J_n(w_n,w_{-n})$ to derive three solution methods—integrated optimization, Gauss-Seidel best-response, and decentralized MARL—and evaluate them on real Germany–Austria data. Contributions: prove equilibrium existence for Γ, reveal its generalized potential nature, and compare performance across methods showing MARL lowers total market cost but increases profit inequality, with coupling reducing costs for large zones. Significance: provides a rigorous framework and scalable, data-driven tools for analyzing zonal market designs and cross-border coupling in European ancillary services.

Abstract

We characterize zonal ancillary market coupling relying on noncooperative game theory. To that purpose, we formulate the ancillary market as a multi-leader single follower bilevel problem, that we subsequently cast as a generalized Nash game with side constraints and nonconvex feasibility sets. We determine conditions for equilibrium existence and show that the game has a generalized potential game structure. To compute market equilibrium, we rely on two exact approaches: an integrated optimization approach and Gauss-Seidel best-response, that we compare against multi-agent deep reinforcement learning. On real data from Germany and Austria, simulations indicate that multi-agent deep reinforcement learning achieves the smallest convergence rate but requires pretraining, while best-response is the slowest. On the economics side, multi-agent deep reinforcement learning results in smaller market costs compared to the exact methods, but at the cost of higher variability in the profit allocation among stakeholders. Further, stronger coupling between zones tends to reduce costs for larger zones.

Figures (4)

• Figure 1: Representation of an ancillary market with two zones, with three producers in the first zone and two producers in the second zone. Constraints are highlighted with arrows.
• Figure 2: Episodic reward for each RL based agent throughout the training. Each episode represents passing through the full market data. We plot the rewards divided by zone, each point represent the average reward during an episode and the shaded region the standard deviation.
• Figure 3: Average cost of running the market, obtained with different algorithms.
• Figure 4: Average cost of running the market with 8 and 4 RL agents, evaluated over different values for the export constraints. Each element on the x-axis corresponds to a couple of export constraints: G stands for Germany and A for Austria, the constraints are in MW.

Theorems & Definitions (9)

• Lemma 1: Slater's Constraint Qualification
• proof
• Definition 1: Nash Equilibrium with Side Constraints, pang_nonconvex_2011
• Proposition 1
• proof
• Theorem 1: Existence of a solution for $\Gamma^p$
• proof
• Proposition 2
• proof