An Offline Multi-Agent Reinforcement Learning Framework for Radio Resource Management

TL;DR

This work tackles radio resource management (RRM) in dynamic wireless networks by proposing an offline multi-agent reinforcement learning (MARL) framework. The authors formulate RRM as a partially observable MDP and develop three offline MARL-CQL variants—centralized, independent, and centralized training with decentralized execution (CTDE)—to learn cooperative scheduling policies without online environment interaction. Empirical results show that these offline schemes outperform conventional baselines on a weighted combination of sum-rate and 5th-percentile tail-rate, with CTDE-MARL-CQL achieving a strong balance between performance and computational efficiency. The study also highlights the pivotal roles of dataset quality and size for convergence and policy effectiveness, underscoring the practical potential of offline MARL for scalable and robust RRM in evolving wireless networks.

Abstract

Offline multi-agent reinforcement learning (MARL) addresses key limitations of online MARL, such as safety concerns, expensive data collection, extended training intervals, and high signaling overhead caused by online interactions with the environment. In this work, we propose an offline MARL algorithm for radio resource management (RRM), focusing on optimizing scheduling policies for multiple access points (APs) to jointly maximize the sum and tail rates of user equipment (UEs). We evaluate three training paradigms: centralized, independent, and centralized training with decentralized execution (CTDE). Our simulation results demonstrate that the proposed offline MARL framework outperforms conventional baseline approaches, achieving over a 15\% improvement in a weighted combination of sum and tail rates. Additionally, the CTDE framework strikes an effective balance, reducing the computational complexity of centralized methods while addressing the inefficiencies of independent training. These results underscore the potential of offline MARL to deliver scalable, robust, and efficient solutions for resource management in dynamic wireless networks.

Figures (7)

• Figure 1: A wireless environment consists of $I$ APs and $J$ UEs. Each UE is associated with only one $AP$, which chooses one of its associated UEs to serve at a time.
• Figure 2: An illustrative comparison between centralized MARL, independent MARL, and centralized training decentralized execution MARL. As shown, C-MARL models a joint Q-function using a single neural network, and the policies are drawn from the joint Q-function. In contrast, in I-MARL and CTDE-MARL, each agent models its Q-function as an independent neural network.
• Figure 3: An illustrative comparison between online MARL and offline MARL. Online MARL utilizes online interaction with the environment to optimize the policies. In contrast, offline MARL exploits offline datasets pre-collected using a behavioral policy. Offline MARL training uses the offline dataset, whereas optimum policies are used for online deployment.
• Figure 4: The sum rate, $5$-percentile rate, and Rscore reported for C-MARL algorithm built on top of both SAC and DQN compared to other benchmark schemes.
• Figure 5: The sum rate, $5$-percentile rate, and Rscore reported for the proposed C-MARL-CQL algorithm built on top of SAC and DQN compared to C-MARL and other benchmark schemes.