A Multi-Agent Reinforcement Learning Testbed for Cognitive Radio Applications

TL;DR

The paper tackles the need for realistic multi-agent training in RF environments by extending the RFRL Gym to support MARL. It implements MARL via Gymnasium and Ray RLlib, enabling centralized training with decentralized execution and flexible agent grouping across cooperative, competitive, and mixed settings. Key contributions include a detailed MARL framework, reward calculations for DSA and jamming scenarios, IQ data generation, enhanced rendering, and five RF test scenarios with multi-algorithm evaluation. The results demonstrate that the environment supports learning across multiple agents and settings, and the authors outline concrete future directions such as hardware integration and richer RF modeling to enhance realism and scalability.

Abstract

Technological trends show that Radio Frequency Reinforcement Learning (RFRL) will play a prominent role in the wireless communication systems of the future. Applications of RFRL range from military communications jamming to enhancing WiFi networks. Before deploying algorithms for these purposes, they must be trained in a simulation environment to ensure adequate performance. For this reason, we previously created the RFRL Gym: a standardized, accessible tool for the development and testing of reinforcement learning (RL) algorithms in the wireless communications space. This environment leveraged the OpenAI Gym framework and featured customizable simulation scenarios within the RF spectrum. However, the RFRL Gym was limited to training a single RL agent per simulation; this is not ideal, as most real-world RF scenarios will contain multiple intelligent agents in cooperative, competitive, or mixed settings, which is a natural consequence of spectrum congestion. Therefore, through integration with Ray RLlib, multi-agent reinforcement learning (MARL) functionality for training and assessment has been added to the RFRL Gym, making it even more of a robust tool for RF spectrum simulation. This paper provides an overview of the updated RFRL Gym environment. In this work, the general framework of the tool is described relative to comparable existing resources, highlighting the significant additions and refactoring we have applied to the Gym. Afterward, results from testing various RF scenarios in the MARL environment and future additions are discussed.

Figures (7)

• Figure 1: A rendering of an abstracted RF spectrum in the multi-agent RFRL Gym.
• Figure 2: Diagram of information flow in the multi-agent RFRL Gym.
• Figure 3: Results from testing Scenario 1. To lessen the impacts of episode reward fluctuations upon the display, an exponentially weighted moving average (EWMA) with $\beta=0.75$ was applied.
• Figure 4: Results from testing Scenario 2. An EWMA with $\beta=0.75$ was used.
• Figure 5: Results from testing Scenario 3. An EWMA with $\beta=0.75$ was used.