Learning to Reflect: Hierarchical Multi-Agent Reinforcement Learning for CSI-Free mmWave Beam-Focusing

TL;DR

This work establishes HMARL as a practical solution for intelligent mmWave environments by eliminating CSI overhead while maintaining high-fidelity beam-focusing by eliminating CSI overhead while maintaining high-fidelity beam-focusing.

Abstract

Reconfigurable Intelligent Surfaces promise to transform wireless environments, yet practical deployment is hindered by the prohibitive overhead of Channel State Information (CSI) estimation and the dimensionality explosion inherent in centralized optimization. This paper proposes a Hierarchical Multi-Agent Reinforcement Learning (HMARL) framework for the control of mechanically reconfigurable reflective surfaces in millimeter-wave (mmWave) systems. We introduce a "CSI-free" paradigm that substitutes pilot-based channel estimation with readily available user localization data. To manage the massive combinatorial action space, the proposed architecture utilizes Multi-Agent Proximal Policy Optimization (MAPPO) under a Centralized Training with Decentralized Execution (CTDE) paradigm. The proposed architecture decomposes the control problem into two abstraction levels: a high-level controller for user-to-reflector allocation and decentralized low-level controllers for low-level focal point optimization. Comprehensive ray-tracing evaluations demonstrate that the framework achieves 2.81-7.94 dB RSSI improvements over centralized baselines, with the performance advantage widening as system complexity increases. Scalability analysis reveals that the system maintains sustained efficiency, exhibiting minimal per-user performance degradation and stable total power utilization even when user density doubles. Furthermore, robustness validation confirms the framework's viability across varying reflector aperture sizes (45-99 tiles) and demonstrates graceful performance degradation under localization errors up to 0.5 m. By eliminating CSI overhead while maintaining high-fidelity beam-focusing, this work establishes HMARL as a practical solution for intelligent mmWave environments.

Figures (13)

• Figure 1: Reflector with hexagonal tiles. Each tile can be rotated in both elevation ($\theta$) and azimuth ($\phi$) angles.
• Figure 2: Temporal coordination between hierarchical controllers. The high-level allocation controller makes high-level user-to-reflector assignment decisions every $T$ timesteps, while low-level controllers continuously optimize focal points at every timestep. This temporal abstraction enables high-level planning at the high level while maintaining rapid low-level adaptation at the low level.
• Figure 3: Hierarchical Multi-Agent Reinforcement Learning Architecture. The high-level controller performs high-level user-to-reflector allocation using global system state at extended temporal intervals ($T$ timesteps), while low-level controllers execute low-level focal point optimization using masked local observations at every timestep. The framework employs Hierarchical Multi-Agent Reinforcement Learning (HMARL), enabling global state access during training while maintaining practical deployment scalability through local observation-based execution. Dashed arrows represent aggregated reward feedback from low-level controllers to the high-level coordinator.
• Figure 4: Experimental setup of a conference room for simulation. The AP is depicted in blue, while the users are shown in green. The users are within the UE region.
• Figure 5: Training performance of the two-user scenario: episode-averaged reward for two-user scenario across four methods over 3,200 episodes.