Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Design of mmWave Initial Access Codebooks using Reinforcement Learning

Sabrine Aroua, Christos Anastasios Bovolis, Bo Göransson, Anastasios Giovanidis, Mathieu Leconte, Apostolos Destounis

TL;DR

This work addresses adaptive initial access in mmWave 5G by learning to design SSB codebooks from a pool of expert beams. Framing the problem as a POMDP, it employs an actor–critic RL policy that sequentially selects $n$ beams from a pool of $m$ beams to maximize UE association during cell search, while leveraging expert designs to ensure safety and compatibility. Empirical results show the Neural Codebook consistently outperforms static expert codebooks and several baselines, achieving up to $>10\%$ gains in user connectivity and higher rediscovery rates across diverse deployments. The approach offers a practical, data-driven pathway to more flexible, resilient beam management in next-generation wireless networks, with potential extensions to direct SNR optimization and multi-metric objectives.

Abstract

Initial access (IA) is the process by which user equipment (UE) establishes its first connection with a base station. In 5G systems, particularly at millimeter-wave frequencies, IA integrates beam management to support highly directional transmissions. The base station employs a codebook of beams for the transmission of Synchronization Signal Blocks (SSBs), which are periodically swept to detect and connect users. The design of this SSB codebook is critical for ensuring reliable, wide-area coverage. In current networks, SSB codebooks are meticulously engineered by domain experts. While these expert-defined codebooks provide a robust baseline, they lack flexibility in dynamic or heterogeneous environments where user distributions vary, limiting their overall effectiveness. This paper proposes a hybrid Reinforcement Learning (RL) framework for adaptive SSB codebook design. Building on top of expert knowledge, the RL agent leverages a pool of expert-designed SSB beams and learns to adaptively select or combine them based on real-time feedback. This enables the agent to dynamically tailor codebooks to the actual environment, without requiring explicit user location information, while always respecting practical beam constraints. Simulation results demonstrate that, on average, the proposed approach improves user connectivity by 10.8$\%$ compared to static expert configurations. These findings highlight the potential of combining expert knowledge with data-driven optimization to achieve more intelligent, flexible, and resilient beam management in next-generation wireless networks.

Adaptive Design of mmWave Initial Access Codebooks using Reinforcement Learning

TL;DR

This work addresses adaptive initial access in mmWave 5G by learning to design SSB codebooks from a pool of expert beams. Framing the problem as a POMDP, it employs an actor–critic RL policy that sequentially selects beams from a pool of beams to maximize UE association during cell search, while leveraging expert designs to ensure safety and compatibility. Empirical results show the Neural Codebook consistently outperforms static expert codebooks and several baselines, achieving up to gains in user connectivity and higher rediscovery rates across diverse deployments. The approach offers a practical, data-driven pathway to more flexible, resilient beam management in next-generation wireless networks, with potential extensions to direct SNR optimization and multi-metric objectives.

Abstract

Initial access (IA) is the process by which user equipment (UE) establishes its first connection with a base station. In 5G systems, particularly at millimeter-wave frequencies, IA integrates beam management to support highly directional transmissions. The base station employs a codebook of beams for the transmission of Synchronization Signal Blocks (SSBs), which are periodically swept to detect and connect users. The design of this SSB codebook is critical for ensuring reliable, wide-area coverage. In current networks, SSB codebooks are meticulously engineered by domain experts. While these expert-defined codebooks provide a robust baseline, they lack flexibility in dynamic or heterogeneous environments where user distributions vary, limiting their overall effectiveness. This paper proposes a hybrid Reinforcement Learning (RL) framework for adaptive SSB codebook design. Building on top of expert knowledge, the RL agent leverages a pool of expert-designed SSB beams and learns to adaptively select or combine them based on real-time feedback. This enables the agent to dynamically tailor codebooks to the actual environment, without requiring explicit user location information, while always respecting practical beam constraints. Simulation results demonstrate that, on average, the proposed approach improves user connectivity by 10.8 compared to static expert configurations. These findings highlight the potential of combining expert knowledge with data-driven optimization to achieve more intelligent, flexible, and resilient beam management in next-generation wireless networks.

Paper Structure

This paper contains 14 sections, 10 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Prior Work
  3. Our Contribution
  4. Network Model and Problem Statement
  5. System Model
  6. Problem Definition
  7. Adaptive Codebook Design with RL
  8. POMDP Formulation
  9. Neural Network Architecture
  10. Training and Deployment
  11. Performance Evaluation
  12. Experimental setup
  13. Results & Discussion
  14. Conclusion

Figures (5)

  • Figure 1: Cell Search (CS) procedure.
  • Figure 2: Example of generated environment with inhomogeneous UEs' deployment.
  • Figure 3: CDF of the fraction of connected/covered UEs.
  • Figure 4: Relative improvement (%) in the number of connected devices with the Neural Codebook over expert codebooks $c_1$ (purple) and $c_2$ (cyan).
  • Figure 5: Top $10\%$ rediscovered UEs' SSB SNR.