Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Structured Reinforcement Learning for Delay-Optimal Data Transmission in Dense mmWave Networks

Shufan Wang, Guojun Xiong, Shichen Zhang, Huacheng Zeng, Jian Li, Shivendra Panwar

TL;DR

The paper tackles delay minimization for data packet transmission in dense mmWave networks by formulating mmDPT as a fairness-constrained RMAB problem (RMAB-F). It develops a low-complexity LP-based mmDPT Index Policy with asymptotic optimality and a structured Thompson sampling algorithm mmDPT-TS that leverages this structure to achieve sublinear Bayesian regret $\tilde{O}(\sqrt{T\log T})$ while remaining computation-efficient. Theoretical guarantees are complemented by extensive experiments on a realistic 60 GHz mmWave testbed and synthetic traces, showing significant gains over baselines and validating the global-attractor assumption. Collectively, the approach enables practical, delay-aware data transmission with fairness in dense mmWave deployments.

Abstract

We study the data packet transmission problem (mmDPT) in dense cell-free millimeter wave (mmWave) networks, i.e., users sending data packet requests to access points (APs) via uplinks and APs transmitting requested data packets to users via downlinks. Our objective is to minimize the average delay in the system due to APs' limited service capacity and unreliable wireless channels between APs and users. This problem can be formulated as a restless multi-armed bandits problem with fairness constraint (RMAB-F). Since finding the optimal policy for RMAB-F is intractable, existing learning algorithms are computationally expensive and not suitable for practical dynamic dense mmWave networks. In this paper, we propose a structured reinforcement learning (RL) solution for mmDPT by exploiting the inherent structure encoded in RMAB-F. To achieve this, we first design a low-complexity and provably asymptotically optimal index policy for RMAB-F. Then, we leverage this structure information to develop a structured RL algorithm called mmDPT-TS, which provably achieves an \tilde{O}(\sqrt{T}) Bayesian regret. More importantly, mmDPT-TS is computation-efficient and thus amenable to practical implementation, as it fully exploits the structure of index policy for making decisions. Extensive emulation based on data collected in realistic mmWave networks demonstrate significant gains of mmDPT-TS over existing approaches.

Structured Reinforcement Learning for Delay-Optimal Data Transmission in Dense mmWave Networks

TL;DR

The paper tackles delay minimization for data packet transmission in dense mmWave networks by formulating mmDPT as a fairness-constrained RMAB problem (RMAB-F). It develops a low-complexity LP-based mmDPT Index Policy with asymptotic optimality and a structured Thompson sampling algorithm mmDPT-TS that leverages this structure to achieve sublinear Bayesian regret while remaining computation-efficient. Theoretical guarantees are complemented by extensive experiments on a realistic 60 GHz mmWave testbed and synthetic traces, showing significant gains over baselines and validating the global-attractor assumption. Collectively, the approach enables practical, delay-aware data transmission with fairness in dense mmWave deployments.

Abstract

We study the data packet transmission problem (mmDPT) in dense cell-free millimeter wave (mmWave) networks, i.e., users sending data packet requests to access points (APs) via uplinks and APs transmitting requested data packets to users via downlinks. Our objective is to minimize the average delay in the system due to APs' limited service capacity and unreliable wireless channels between APs and users. This problem can be formulated as a restless multi-armed bandits problem with fairness constraint (RMAB-F). Since finding the optimal policy for RMAB-F is intractable, existing learning algorithms are computationally expensive and not suitable for practical dynamic dense mmWave networks. In this paper, we propose a structured reinforcement learning (RL) solution for mmDPT by exploiting the inherent structure encoded in RMAB-F. To achieve this, we first design a low-complexity and provably asymptotically optimal index policy for RMAB-F. Then, we leverage this structure information to develop a structured RL algorithm called mmDPT-TS, which provably achieves an \tilde{O}(\sqrt{T}) Bayesian regret. More importantly, mmDPT-TS is computation-efficient and thus amenable to practical implementation, as it fully exploits the structure of index policy for making decisions. Extensive emulation based on data collected in realistic mmWave networks demonstrate significant gains of mmDPT-TS over existing approaches.
Paper Structure (24 sections, 5 theorems, 49 equations, 10 figures, 1 table, 2 algorithms)

This paper contains 24 sections, 5 theorems, 49 equations, 10 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. System model and Problem Formulation
  3. System Model
  4. MDP-based Problem Formulation
  5. Index Policy Design and Analysis
  6. The Relaxed Problem
  7. The mmDT Index Policy
  8. Asymptotic Optimality
  9. Structured Reinforcement Learning
  10. Structured RL Algorithm: mmDPT-TS
  11. The Learning Regret
  12. Proof Sketch of Theorem \ref{['thm:regret']}
  13. Experiments
  14. Evaluation Setup
  15. Evaluation Results
  16. ...and 9 more sections

Key Result

Theorem 1

Our mmDPT Index Policy$\pi^*$ is asymptotically optimal under Definition assump:global_attractor, i.e., $\lim_{\rho\rightarrow\infty}V_{\pi^*}^\rho-V_{\pi^{opt}}^\rho=0.$

Figures (10)

  • Figure 1: A dense mmWave network in a small conference room, where the dashed lines indicate communications between APs and users, i.e., users sending data packet requests to APs and APs transmitting real data packets to users. See Section \ref{['sec:exp']} for more details on our mmWave testbed.
  • Figure 2: Measurement setup and experiment scenario for data packet transmission in dense mmWave networks.
  • Figure 3: The measured error vector magnitude (EVM) of decoded signal constellations at users which receive data packet from APs (via downlinks) in our mmWave testbed (See Section \ref{['sec:exp']}). The three curves correspond to the three transmissions in Figure \ref{['fig:example']}.
  • Figure 4: Asymptotic optimality: 60GHz mmWave testbed.
  • Figure 5: Asymptotic optimality: Synthetic data traces.
  • ...and 5 more figures

Theorems & Definitions (12)

  • Definition 1
  • Remark 1
  • Definition 1
  • Theorem 1
  • Remark 2
  • Theorem 2
  • Lemma 1
  • Remark 3
  • Lemma 2
  • proof
  • ...and 2 more