Dynamics of Resource Allocation in O-RANs: An In-depth Exploration of On-Policy and Off-Policy Deep Reinforcement Learning for Real-Time Applications
Manal Mehdaoui, Amine Abouaomar
TL;DR
The paper addresses resource allocation in Open Radio Access Networks (O-RAN) under Quality-of-Service constraints for latency-sensitive and latency-tolerant users. It compares two Deep Reinforcement Learning (DRL) approaches—on-policy Proximal Policy Optimization (PPO) and off-policy Advantage Actor-Critic with Experience Replay (ACER)—formulated as a Markov Decision Process with state $s(t)$, action $a(t)$, and reward $R(s,a) = -\alpha \sum_m P_t^m$, against a Mixed Integer Programming (MIP) baseline. Using the Alibaba cluster-trace-v2018 dataset and a replication setup, the study shows that both PPO and ACER outperform a greedy baseline, with PPO achieving a favorable energy–latency balance and ACER exhibiting faster convergence, albeit with sensitivity to network size. The results reinforce the practicality of DRL-based cross-slice resource orchestration in dynamic O-RAN environments and demonstrate reproducibility relative to prior work. These insights contribute to designing real-time, energy-efficient resource allocation strategies for dynamic slicing in O-RAN systems.
Abstract
Deep Reinforcement Learning (DRL) is a powerful tool used for addressing complex challenges in mobile networks. This paper investigates the application of two DRL models, on-policy and off-policy, in the field of resource allocation for Open Radio Access Networks (O-RAN). The on-policy model is the Proximal Policy Optimization (PPO), and the off-policy model is the Sample Efficient Actor-Critic with Experience Replay (ACER), which focuses on resolving the challenges of resource allocation associated with a Quality of Service (QoS) application that has strict requirements. Motivated by the original work of Nessrine Hammami and Kim Khoa Nguyen, this study is a replication to validate and prove the findings. Both PPO and ACER are used within the same experimental setup to assess their performance in a scenario of latency-sensitive and latency-tolerant users and compare them. The aim is to verify the efficacy of on-policy and off-policy DRL models in the context of O-RAN resource allocation. Results from this replication contribute to the ongoing scientific research and offer insights into the reproducibility and generalizability of the original research. This analysis reaffirms that both on-policy and off-policy DRL models have better performance than greedy algorithms in O-RAN settings. In addition, it confirms the original observations that the on-policy model (PPO) gives a favorable balance between energy consumption and user latency, while the off-policy model (ACER) shows a faster convergence. These findings give good insights to optimize resource allocation strategies in O-RANs. Index Terms: 5G, O-RAN, resource allocation, ML, DRL, PPO, ACER.