Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Asynchronous MultiAgent Reinforcement Learning for 5G Routing under Side Constraints

Sebastian Racedo, Brigitte Jaumard, Oscar Delgado, Meysam Masoudi

TL;DR

The paper addresses real-time, QoS-constrained routing in 5G-like networks by formulating it as a multi-commodity routing problem with per-service constraints. It introduces AMARL, an asynchronous multi-agent PPO framework with one agent per service that operates on local episodic copies and commits resource deltas to a shared environment via an action-masking mechanism, enabling feasibility under shared capacity and latency constraints. Experiments on an O-RAN–inspired Montreal network show AMARL achieves GoS and end-to-end latency on par with a strong centralized PPO baseline while delivering ~30% faster training and ~15% faster evaluation, highlighting practical gains in wall-clock performance. The approach offers fault isolation, service-level specialization, and modular scalability suitable for near-real-time xApps in O-RAN and potentially broader distributed routing domains.

Abstract

Networks in the current 5G and beyond systems increasingly carry heterogeneous traffic with diverse quality-of-service constraints, making real-time routing decisions both complex and time-critical. A common approach, such as a heuristic with human intervention or training a single centralized RL policy or synchronizing updates across multiple learners, struggles with scalability and straggler effects. We address this by proposing an asynchronous multi-agent reinforcement learning (AMARL) framework in which independent PPO agents, one per service, plan routes in parallel and commit resource deltas to a shared global resource environment. This coordination by state preserves feasibility across services and enables specialization for service-specific objectives. We evaluate the method on an O-RAN like network simulation using nearly real-time traffic data from the city of Montreal. We compared against a single-agent PPO baseline. AMARL achieves a similar Grade of Service (acceptance rate) (GoS) and end-to-end latency, with reduced training wall-clock time and improved robustness to demand shifts. These results suggest that asynchronous, service-specialized agents provide a scalable and practical approach to distributed routing, with applicability extending beyond the O-RAN domain.

Asynchronous MultiAgent Reinforcement Learning for 5G Routing under Side Constraints

TL;DR

The paper addresses real-time, QoS-constrained routing in 5G-like networks by formulating it as a multi-commodity routing problem with per-service constraints. It introduces AMARL, an asynchronous multi-agent PPO framework with one agent per service that operates on local episodic copies and commits resource deltas to a shared environment via an action-masking mechanism, enabling feasibility under shared capacity and latency constraints. Experiments on an O-RAN–inspired Montreal network show AMARL achieves GoS and end-to-end latency on par with a strong centralized PPO baseline while delivering ~30% faster training and ~15% faster evaluation, highlighting practical gains in wall-clock performance. The approach offers fault isolation, service-level specialization, and modular scalability suitable for near-real-time xApps in O-RAN and potentially broader distributed routing domains.

Abstract

Networks in the current 5G and beyond systems increasingly carry heterogeneous traffic with diverse quality-of-service constraints, making real-time routing decisions both complex and time-critical. A common approach, such as a heuristic with human intervention or training a single centralized RL policy or synchronizing updates across multiple learners, struggles with scalability and straggler effects. We address this by proposing an asynchronous multi-agent reinforcement learning (AMARL) framework in which independent PPO agents, one per service, plan routes in parallel and commit resource deltas to a shared global resource environment. This coordination by state preserves feasibility across services and enables specialization for service-specific objectives. We evaluate the method on an O-RAN like network simulation using nearly real-time traffic data from the city of Montreal. We compared against a single-agent PPO baseline. AMARL achieves a similar Grade of Service (acceptance rate) (GoS) and end-to-end latency, with reduced training wall-clock time and improved robustness to demand shifts. These results suggest that asynchronous, service-specialized agents provide a scalable and practical approach to distributed routing, with applicability extending beyond the O-RAN domain.
Paper Structure (18 sections, 2 equations, 9 figures, 5 tables)

This paper contains 18 sections, 2 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. General MARL
  4. Asynchronicity
  5. RL in networking
  6. Problem Formulation
  7. Asynchronous Multi-Agent Design
  8. Agents, Environment, and Asynchrony
  9. Observations, Actions and Mask
  10. Reward Structure
  11. Experimental Setup
  12. Network Topology
  13. Results
  14. Metrics and Baseline
  15. Learning Dynamics and Wall-Clock
  16. ...and 3 more sections

Figures (9)

  • Figure 1: Single Agent Diagram
  • Figure 2: Asynchronous Multi-Agent Diagram
  • Figure 3: Example of step where the current segment is 0 (src is RU1 and target is DU1)
  • Figure 4: Fixed placement of logical functions capable of accommodating a dynamic traffic over 24 hours
  • Figure 5: Learning curves for the AMARL and SARL during training.
  • ...and 4 more figures