Adaptive Passive Beamforming in RIS-Aided Communications With Q-Learning

TL;DR

This work tackles RIS-aided wireless optimization without CSI by framing RIS configuration as a stochastic shortest path problem and solving it with Q-learning. An adaptive protocol tests RIS codewords from two codebooks, using feedback to progressively concentrate on the most promising configurations while minimizing pilot overhead. The approach leverages a Bayes-optimal classifier and an MDP/SSP framework to derive an acquisition policy that reduces configuration trials compared to hierarchical or exhaustive searches. Numerical results show that the Q-learning strategy achieves near-optimal channel strength with significantly fewer pilots, highlighting its practical impact for CSI-free RIS beamforming. Future work points to deep learning to scale the state space and further improve efficiency as codebook sizes grow.

Abstract

Reconfigurable Intelligent Surfaces (RIS) appear as a promising solution to combat wireless channel fading and interferences. However, the elements of the RIS need to be properly oriented to boost the data transmission rate. In this work, we propose a new strategy to adaptively configure the RIS without Channel State Information (CSI). Our goal is to minimize the number of RIS configurations to be tested to find the optimal one. We formulate the problem as a stochastic shortest path problem, and use Q-Learning to solve it.

Figures (6)

• Figure 1: Setup
• Figure 2: Adaptive Protocol
• Figure 3: Stochastic shortest path problem
• Figure 4: Training Procedure of Q-Learning
• Figure 5: Evolution of the average length $L_{min}$ during the different epochs of the Q-Learning