No Free Lunch: Balancing Learning and Exploitation at the Network Edge

TL;DR

This work tackles the cost of learning for reinforcement learning-based network optimization at the edge, where training and data transfer compete with user traffic. It introduces a cost-of-learning meta-model that splits each episode into exploitation and update phases, and proposes two strategies: a constant update duration and an adaptive duration that stops learning once convergence is detected. Through a network-slicing uplink use-case, the authors show that ignoring training costs yields inflated performance estimates, and that adaptive training durations (e.g., $T_{\rho}=4$) can achieve a favorable balance between convergence speed and user QoS. The findings underscore the need for cost-aware design in online edge learning and motivate future work in hierarchical and meta-learning approaches for broader applicability.

Abstract

Over the last few years, the DRL paradigm has been widely adopted for 5G and beyond network optimization because of its extreme adaptability to many different scenarios. However, collecting and processing learning data entail a significant cost in terms of communication and computational resources, which is often disregarded in the networking literature. In this work, we analyze the cost of learning in a resource-constrained system, defining an optimization problem in which training a DRL agent makes it possible to improve the resource allocation strategy but also reduces the number of available resources. Our simulation results show that the cost of learning can be critical when evaluating DRL schemes on the network edge and that assuming a cost-free learning model can lead to significantly overestimating performance.

Figures (4)

• Figure 1: Mean performance over time with fixed $T_{\rho}$.
• Figure 2: Boxplots of the performance with fixed $T_{\rho}$.
• Figure 3: Mean performance over time with adaptive $T_{\rho}$.
• Figure 4: Boxplots of the performance with adaptive $T_{\rho}$.