Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cooperative Hierarchical Deep Reinforcement Learning based Joint Sleep and Power Control in RIS-aided Energy-Efficient RAN

Hao Zhou, Medhat Elsayed, Majid Bavand, Raimundas Gaigalas, Steve Furr, Melike Erol-Kantarci

TL;DR

The paper tackles energy efficiency in RIS-aided RANs by addressing sleep control (long-term) and power control (short-term) on different timescales. It introduces Cooperative Hierarchical Deep Reinforcement Learning (Co-HDRL) with a cross-entropy–enabled meta-controller and correlated-equilibrium–based sub-controllers, complemented by a fractional programming RIS phase-shift optimization. The approach demonstrates that RIS-assisted sleep control can substantially reduce energy consumption and boost EE, with FP-based RIS optimization delivering fast convergence and superior performance over surrogate methods. The work highlights practical implications for 6G/O-RAN deployments, offering a scalable, stable framework for multi-timescale network management and RIS configuration; future work may explore RIS placement and more advanced metasurface designs.

Abstract

Energy efficiency (EE) is one of the most important metrics for envisioned 6G networks, and sleep control, as a cost-efficient approach, can significantly lower power consumption by switching off network devices selectively. Meanwhile, the reconfigurable intelligent surface (RIS) has emerged as a promising technique to enhance the EE of future wireless networks. In this work, we jointly consider sleep and transmission power control for RIS-aided energy-efficient networks. In particular, considering the timescale difference between sleep control and power control, we introduce a cooperative hierarchical deep reinforcement learning (Co-HDRL) algorithm, enabling hierarchical and intelligent decision-making. Specifically, the meta-controller in Co-HDRL uses cross-entropy metrics to evaluate the policy stability of sub-controllers, and sub-controllers apply the correlated equilibrium to select optimal joint actions. Compared with conventional HDRL, Co-HDRL enables more stable high-level policy generations and low-level action selections. Then, we introduce a fractional programming method for RIS phase-shift control, maximizing the sum-rate under a given transmission power. In addition, we proposed a low-complexity surrogate optimization method as a baseline for RIS control. Finally, simulations show that the RIS-assisted sleep control can achieve more than 16\% lower energy consumption and 30\% higher EE than baseline algorithms.

Cooperative Hierarchical Deep Reinforcement Learning based Joint Sleep and Power Control in RIS-aided Energy-Efficient RAN

TL;DR

The paper tackles energy efficiency in RIS-aided RANs by addressing sleep control (long-term) and power control (short-term) on different timescales. It introduces Cooperative Hierarchical Deep Reinforcement Learning (Co-HDRL) with a cross-entropy–enabled meta-controller and correlated-equilibrium–based sub-controllers, complemented by a fractional programming RIS phase-shift optimization. The approach demonstrates that RIS-assisted sleep control can substantially reduce energy consumption and boost EE, with FP-based RIS optimization delivering fast convergence and superior performance over surrogate methods. The work highlights practical implications for 6G/O-RAN deployments, offering a scalable, stable framework for multi-timescale network management and RIS configuration; future work may explore RIS placement and more advanced metasurface designs.

Abstract

Energy efficiency (EE) is one of the most important metrics for envisioned 6G networks, and sleep control, as a cost-efficient approach, can significantly lower power consumption by switching off network devices selectively. Meanwhile, the reconfigurable intelligent surface (RIS) has emerged as a promising technique to enhance the EE of future wireless networks. In this work, we jointly consider sleep and transmission power control for RIS-aided energy-efficient networks. In particular, considering the timescale difference between sleep control and power control, we introduce a cooperative hierarchical deep reinforcement learning (Co-HDRL) algorithm, enabling hierarchical and intelligent decision-making. Specifically, the meta-controller in Co-HDRL uses cross-entropy metrics to evaluate the policy stability of sub-controllers, and sub-controllers apply the correlated equilibrium to select optimal joint actions. Compared with conventional HDRL, Co-HDRL enables more stable high-level policy generations and low-level action selections. Then, we introduce a fractional programming method for RIS phase-shift control, maximizing the sum-rate under a given transmission power. In addition, we proposed a low-complexity surrogate optimization method as a baseline for RIS control. Finally, simulations show that the RIS-assisted sleep control can achieve more than 16\% lower energy consumption and 30\% higher EE than baseline algorithms.
Paper Structure (24 sections, 22 equations, 7 figures, 1 table, 2 algorithms)

This paper contains 24 sections, 22 equations, 7 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Network and System Model
  4. RIS-Assisted Channel Model
  5. BS Energy Consumption Model
  6. Problem Formulation
  7. Cooperative Hierarchical Deep Reinforcement Learning for Sleep Control and Power Control
  8. Hierarchical SMDP
  9. SMDP Definition of Meta-controller and Sub-controllers
  10. Co-HDRL Framework and Network Training
  11. High-level Cross-entropy enabled Meta-controller Policy
  12. Low-level Sub-controllers Cooperation
  13. Computational Complexity Analyses
  14. Baseline: Hierarchical Deep Reinforcement Learning
  15. RIS Phase-shift Optimization
  16. ...and 9 more sections

Figures (7)

  • Figure 1: RIS-aided heterogeneous network.
  • Figure 2: MDP and SMDP comparison.
  • Figure 3: Overall architecture of the proposed Co-HDRL.
  • Figure 4: Co-HDRL system update between meta-controller and sub-controllers.
  • Figure 5: Performance comparison of sleep control and RIS.
  • ...and 2 more figures