A Novel Multiple Access Scheme for Heterogeneous Wireless Communications using Symmetry-aware Continual Deep Reinforcement Learning

TL;DR

The paper tackles non-stationary, heterogeneous multi-channel MAC for Metaverse services by introducing a symmetry-aware continual learning mechanism embedded in a Double-Dueling Deep Q-Learning (D3QL) MAC agent. The approach enables an intelligent agent to coexist with legacy TDMA/CSMA/CH UEs while adaptively selecting channel and packet length under a finite-state, context-driven framework, with fairness enforced via a water-filling target. Key contributions include a MINLP problem formulation under POMDP conditions, a detailed D3QL-based solution with LSTM-based approximators, a symmetry-aware continual learning scheme that caps context explosion, and theoretical efficiency bounds on context counts. Empirical results demonstrate improved throughput, reduced collisions, and enhanced fairness across both fixed and stochastic context transitions, supporting the potential of CL-enabled DRL for self-sustaining 6G networks in Metaverse scenarios.

Abstract

The Metaverse holds the potential to revolutionize digital interactions through the establishment of a highly dynamic and immersive virtual realm over wireless communications systems, offering services such as massive twinning and telepresence. This landscape presents novel challenges, particularly efficient management of multiple access to the frequency spectrum, for which numerous adaptive Deep Reinforcement Learning (DRL) approaches have been explored. However, challenges persist in adapting agents to heterogeneous and non-stationary wireless environments. In this paper, we present a novel approach that leverages Continual Learning (CL) to enhance intelligent Medium Access Control (MAC) protocols, featuring an intelligent agent coexisting with legacy User Equipments (UEs) with varying numbers, protocols, and transmission profiles unknown to the agent for the sake of backward compatibility and privacy. We introduce an adaptive Double and Dueling Deep Q-Learning (D3QL)-based MAC protocol, enriched by a symmetry-aware CL mechanism, which maximizes intelligent agent throughput while ensuring fairness. Mathematical analysis validates the efficiency of our proposed scheme, showcasing superiority over conventional DRL-based techniques in terms of throughput, collision rate, and fairness, coupled with real-time responsiveness in highly dynamic scenarios.

Figures (12)

• Figure 1: A lakeside holographic meeting room in the Metaverse enabled by a cloud-network integrated infrastructure powered by technologies including deterministic networking, time-sensitive networking, and intelligent medium access control, with end users connected via 6G, 5G, WiFi, and fiber connections.
• Figure 2: The system model.
• Figure 3: A scenario in which the agent initiates the transmission of packets with lengths $5$ and $2$ at time slots $t = 2$ and $t = 9$, respectively. The values of the decision variable $r$, along with the support variables $d$, $z$, and $m$, can be compared accordingly. At the start of transmissions, both $r$ and $z$ equal the packet length, but $z$ decrements each time slot to enable the calculation of the per time slot transmission indicator $m$.
• Figure 4: The D3QL agent.
• Figure 5: The evaluation network of D3QL Agent (Fig. \ref{['fig_d3ql']}).