Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MARLEM: A Multi-Agent Reinforcement Learning Simulation Framework for Implicit Cooperation in Decentralized Local Energy Markets

Nelson Salazar-Pena, Alejandra Tabares, Andres Gonzalez-Mancera

TL;DR

This work introduces MARLEM, an open-source, Gymnasium-compatible MARL framework for studying implicit cooperation in Local Energy Markets modeled as a Dec-POMDP. It unifies a modular market with realistic grid constraints and a comprehensive analytics suite to analyze emergent coordination without explicit communication. The framework embeds system-level KPIs into agent observations and rewards to encourage collectively beneficial behavior under fully decentralized training and execution (DTDE). A battery-storage case study demonstrates that strategic storage can stabilize prices and improve grid coordination, validating the framework’s fidelity and its potential to inform the design of future decentralized, intelligent energy systems. Future work includes extensive MARL validation across DTDE/CTDE/CTCE paradigms, richer DER models, scalability analyses, and calibration with real-world data.

Abstract

This paper introduces a novel, open-source MARL simulation framework for studying implicit cooperation in LEMs, modeled as a decentralized partially observable Markov decision process and implemented as a Gymnasium environment for MARL. Our framework features a modular market platform with plug-and-play clearing mechanisms, physically constrained agent models (including battery storage), a realistic grid network, and a comprehensive analytics suite to evaluate emergent coordination. The main contribution is a novel method to foster implicit cooperation, where agents' observations and rewards are enhanced with system-level key performance indicators to enable them to independently learn strategies that benefit the entire system and aim for collectively beneficial outcomes without explicit communication. Through representative case studies (available in a dedicated GitHub repository in https://github.com/salazarna/marlem, we show the framework's ability to analyze how different market configurations (such as varying storage deployment) impact system performance. This illustrates its potential to facilitate emergent coordination, improve market efficiency, and strengthen grid stability. The proposed simulation framework is a flexible, extensible, and reproducible tool for researchers and practitioners to design, test, and validate strategies for future intelligent, decentralized energy systems.

MARLEM: A Multi-Agent Reinforcement Learning Simulation Framework for Implicit Cooperation in Decentralized Local Energy Markets

TL;DR

This work introduces MARLEM, an open-source, Gymnasium-compatible MARL framework for studying implicit cooperation in Local Energy Markets modeled as a Dec-POMDP. It unifies a modular market with realistic grid constraints and a comprehensive analytics suite to analyze emergent coordination without explicit communication. The framework embeds system-level KPIs into agent observations and rewards to encourage collectively beneficial behavior under fully decentralized training and execution (DTDE). A battery-storage case study demonstrates that strategic storage can stabilize prices and improve grid coordination, validating the framework’s fidelity and its potential to inform the design of future decentralized, intelligent energy systems. Future work includes extensive MARL validation across DTDE/CTDE/CTCE paradigms, richer DER models, scalability analyses, and calibration with real-world data.

Abstract

This paper introduces a novel, open-source MARL simulation framework for studying implicit cooperation in LEMs, modeled as a decentralized partially observable Markov decision process and implemented as a Gymnasium environment for MARL. Our framework features a modular market platform with plug-and-play clearing mechanisms, physically constrained agent models (including battery storage), a realistic grid network, and a comprehensive analytics suite to evaluate emergent coordination. The main contribution is a novel method to foster implicit cooperation, where agents' observations and rewards are enhanced with system-level key performance indicators to enable them to independently learn strategies that benefit the entire system and aim for collectively beneficial outcomes without explicit communication. Through representative case studies (available in a dedicated GitHub repository in https://github.com/salazarna/marlem, we show the framework's ability to analyze how different market configurations (such as varying storage deployment) impact system performance. This illustrates its potential to facilitate emergent coordination, improve market efficiency, and strengthen grid stability. The proposed simulation framework is a flexible, extensible, and reproducible tool for researchers and practitioners to design, test, and validate strategies for future intelligent, decentralized energy systems.
Paper Structure (53 sections, 11 equations, 7 figures, 2 tables, 1 algorithm)

This paper contains 53 sections, 11 equations, 7 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation
  3. Problem statement
  4. Research gap
  5. Novelty and contribution
  6. Paper structure
  7. Literature review
  8. Centralized and decentralized energy markets
  9. Multi-agent systems in energy grid management
  10. Application and limitations of MARL for DER coordination
  11. Simulation frameworks for LEMs and MARL
  12. Research gap and contribution opportunity
  13. Methodology
  14. Simulation framework architecture
  15. Formulation of the decentralized partially observable Markov decision process
  16. ...and 38 more sections

Figures (7)

  • Figure 1: A high-level class diagram illustrating the relationships among the MARLEM framework's core components.
  • Figure 2: Diagram illustrating the multi-stage order matching and market clearing processes.
  • Figure 3: A sequence diagram for the simulation workflow of the MARLEM framework.
  • Figure 4: Statistical distributions of market clearing prices and trade volumes for the case study.
  • Figure 5: Grid balance deviation over time for the case study.
  • ...and 2 more figures