Multi-Agent Reinforcement Learning Scheduling to Support Low Latency in Teleoperated Driving

TL;DR

This work tackles the stringent end-to-end latency demands of teleoperated driving by applying PPO-based multi-agent reinforcement learning to RAN-level scheduling. It formalizes the problem as a dec-POMDP with per-UE priority actions and a latency-focused reward, and compares IPPO and MAPPO under proportional and greedy allocation schemes. The key finding is that MAPPO combined with greedy allocation delivers the best latency performance, especially as the number of vehicles grows, due to coherent coordination among agents; however, this can come at the expense of fairness. The study demonstrates that intelligent, network-level resource management can meet TD QoS without compromising data integrity, offering a scalable approach for future 5G/6G vehicular networks, with extensions planned to incorporate data quality and energy metrics.

Abstract

The teleoperated driving (TD) scenario comes with stringent Quality of Service (QoS) communication constraints, especially in terms of end-to-end (E2E) latency and reliability. In this context, Predictive Quality of Service (PQoS), possibly combined with Reinforcement Learning (RL) techniques, is a powerful tool to estimate QoS degradation and react accordingly. For example, an intelligent agent can be trained to select the optimal compression configuration for automotive data, and reduce the file size whenever QoS conditions deteriorate. However, compression may inevitably compromise data quality, with negative implications for the TD application. An alternative strategy involves operating at the Radio Access Network (RAN) level to optimize radio parameters based on current network conditions, while preserving data quality. In this paper, we propose Multi-Agent Reinforcement Learning (MARL) scheduling algorithms, based on Proximal Policy Optimization (PPO), to dynamically and intelligently allocate radio resources to minimize E2E latency in a TD scenario. We evaluate two training paradigms, i.e., decentralized learning with local observations (IPPO) vs. centralized aggregation (MAPPO), in conjunction with two resource allocation strategies, i.e., proportional allocation (PA) and greedy allocation (GA). We prove via ns-3 simulations that MAPPO, combined with GA, achieves the best results in terms of latency, especially as the number of vehicles increases.

Figures (6)

• Figure 1: Average reward at the end of the training for and , combined with PA or GA, for $N \in \{3, 5,8\}$.
• Figure 2: Average reward at the beginning (top) and at end of the training (bottom) for , with $N = 5$ .
• Figure 3: Average latency, reward and latency-success probability vs. $N$ at the end of the training, for $(q, c) = (8, 0)$ and $\tau$ = 25 ms.
• Figure 4: The 95-percentile of the latency-violation probability vs. $N$.
• Figure 5: Average latency, reward and latency-success probability vs. the compression configuration at the end of the training, for $N$ = 5 and $\tau$ = 25 ms.