Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Reinforcement Learning Based Block Coordinate Descent for Downlink Weighted Sum-rate Maximization on AI-Native Wireless Networks

Siya Chen, Chee Wei Tan, H. Vincent Poor

TL;DR

A deep reinforcement learning-based block coordinate descent (DRL-based BCD) algorithm to address the nonconvex weighted sum-rate maximization (WSRM) problem with a total power constraint and demonstrates substantial advantages in effectiveness, efficiency, robustness, and interpretability for maximizing sum rates.

Abstract

This paper introduces a deep reinforcement learning-based block coordinate descent (DRL-based BCD) algorithm to address the nonconvex weighted sum-rate maximization (WSRM) problem with a total power constraint. Firstly, we present an efficient block coordinate descent (BCD) method to solve the problem. We then integrate deep reinforcement learning (DRL) techniques into the BCD method and propose the DRL-based BCD algorithm. This approach combines the data-driven learning capability of machine learning techniques with the navigational and decision-making characteristics of the optimization-theoretic-based BCD method. This combination significantly improves the algorithm's performance by reducing its sensitivity to initial points and mitigating the risk of entrapment in local optima. The primary advantages of the proposed DRL-based BCD algorithm lie in its ability to adhere to the constraints of the WSRM problem and significantly enhance accuracy, potentially achieving the exact optimal solution. Moreover, unlike many pure machine-learning approaches, the DRL-based BCD algorithm capitalizes on the underlying theoretical analysis of the WSRM problem's structure. This enables it to be easily trained and computationally efficient while maintaining a level of interpretability. Through numerical experiments, the DRL-based BCD algorithm demonstrates substantial advantages in effectiveness, efficiency, robustness, and interpretability for maximizing sum rates, which also provides valuable potential for designing resource-constrained AI-native wireless optimization strategies in next-generation wireless networks.

Deep Reinforcement Learning Based Block Coordinate Descent for Downlink Weighted Sum-rate Maximization on AI-Native Wireless Networks

TL;DR

A deep reinforcement learning-based block coordinate descent (DRL-based BCD) algorithm to address the nonconvex weighted sum-rate maximization (WSRM) problem with a total power constraint and demonstrates substantial advantages in effectiveness, efficiency, robustness, and interpretability for maximizing sum rates.

Abstract

This paper introduces a deep reinforcement learning-based block coordinate descent (DRL-based BCD) algorithm to address the nonconvex weighted sum-rate maximization (WSRM) problem with a total power constraint. Firstly, we present an efficient block coordinate descent (BCD) method to solve the problem. We then integrate deep reinforcement learning (DRL) techniques into the BCD method and propose the DRL-based BCD algorithm. This approach combines the data-driven learning capability of machine learning techniques with the navigational and decision-making characteristics of the optimization-theoretic-based BCD method. This combination significantly improves the algorithm's performance by reducing its sensitivity to initial points and mitigating the risk of entrapment in local optima. The primary advantages of the proposed DRL-based BCD algorithm lie in its ability to adhere to the constraints of the WSRM problem and significantly enhance accuracy, potentially achieving the exact optimal solution. Moreover, unlike many pure machine-learning approaches, the DRL-based BCD algorithm capitalizes on the underlying theoretical analysis of the WSRM problem's structure. This enables it to be easily trained and computationally efficient while maintaining a level of interpretability. Through numerical experiments, the DRL-based BCD algorithm demonstrates substantial advantages in effectiveness, efficiency, robustness, and interpretability for maximizing sum rates, which also provides valuable potential for designing resource-constrained AI-native wireless optimization strategies in next-generation wireless networks.
Paper Structure (28 sections, 3 theorems, 54 equations, 10 figures, 4 tables, 3 algorithms)

This paper contains 28 sections, 3 theorems, 54 equations, 10 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. System Model and Problem Formulation
  3. Optimization-theoretic-based Algorithm for downlink sum-rate maximization
  4. DRL-based BCD for Downlink Sum-rate Maximization
  5. Overview of DRL for Continuous Control
  6. The DRL-based BCD model
  7. Training Process
  8. Complexity Analysis
  9. Application Extension for Downlink WSRM Beamforming
  10. Numerical Examples
  11. Performance of DRL-based BCD for Downlink WSRM with Fixed Precoders
  12. Samples Generation
  13. Model Configuration
  14. Algorithm Results
  15. Comparison with DRL-based BCD Using Sum Rate as Reward Function
  16. ...and 13 more sections

Key Result

Lemma 1

The optimal solution $\Tilde{\bm{\gamma}}^*$ of eq:sumrate_linear can be expressed as follows: where $\mathbf{B} = \mathbf{F}+\left(1 / \Bar{P}\right) \bm{\sigma}\mathbf{m}^{\top}$, and $z^*_l$ satisfies Moreover, $z^*_l$ can be obtained by the following iteration: Furthermore, the optimal Lagrangian dual $\lambda^*$ of eq:sumrate_linear is given by $\lambda^* = \sum_l w_ly_{l2}^{(t)}$.

Figures (10)

  • Figure 1: An overview of the relationship between the WSRM problem and the reinforcement learning based block coordinate descent algorithm.
  • Figure 2: Evolution of the data rate for $3$ users using the BCD method.
  • Figure 3: The architecture of our DRL-based BCD algorithm for solving \ref{['eq:sumrate_pro3']} includes state information from system parameters ($\mathbf{G}$, $\mathbf{v}$, $\mathbf{w}$, and $\Bar{P}$), the data rate $\Tilde{\bm{\gamma}}^{(t)}$ derived from the BCD algorithm for subproblems, and features of the problem's structure (including $F\mathbf{p}(\tilde{\bm{\gamma}})+\bm{\sigma}$ and $\textup{diag}(\mathbf{w})\log(1+\tilde{\gamma})$). The policy network outputs action $\bm{a}^{(t)}$ based on $\bm{s}^{(t)}$, and the agent receives reward $r^{(t)}$. Then the value network evaluates $\bm{a}^{(t)}$ in $\bm{s}^{(t)}$. This iteration continues until the agent converges to an optimal solution of \ref{['eq:sumrate_pro3']}.
  • Figure 4: Illustration with error bars on the performance of cumulative reward, sum rate error, and convergence steps for the TD3-based BCD model (a-c) and the DDPG-based model (d-f) in Algorithm \ref{['alg:alg2']} in solving WSRM problems.
  • Figure 5: Illustration with error bars on the performance of cumulative reward, sum rate error, and convergence steps for the TD3-based BCD model (a-c) and the DDPG-based model (d-f) using sum rate as the reward function in solving WSRM problems.
  • ...and 5 more figures

Theorems & Definitions (6)

  • Lemma 1
  • Lemma 2
  • Remark 1
  • Example 1
  • Remark 2
  • Lemma 3