Deep reinforcement learning approach to MIMO precoding problem: Optimality and Robustness

TL;DR

A deep reinforcement learning (RL)-based precoding framework that can be used to learn an optimal precoding policy for complex multiple-input multiple-output (MIMO) precoding problems is proposed and the numerical results confirm the effectiveness and robustness of the DRL-based framework.

Abstract

In this paper, we propose a deep reinforcement learning (RL)-based precoding framework that can be used to learn an optimal precoding policy for complex multiple-input multiple-output (MIMO) precoding problems. We model the precoding problem for a single-user MIMO system as an RL problem in which a learning agent sequentially selects the precoders to serve the environment of MIMO system based on contextual information about the environmental conditions, while simultaneously adapting the precoder selection policy based on the reward feedback from the environment to maximize a numerical reward signal. We develop the RL agent with two canonical deep RL (DRL) algorithms, namely deep Q-network (DQN) and deep deterministic policy gradient (DDPG). To demonstrate the optimality of the proposed DRL-based precoding framework, we explicitly consider a simple MIMO environment for which the optimal solution can be obtained analytically and show that DQN- and DDPG-based agents can learn the near-optimal policy to map the environment state of MIMO system to a precoder that maximizes the reward function, respectively, in the codebook-based and non-codebook based MIMO precoding systems. Furthermore, to investigate the robustness of DRL-based precoding framework, we examine the performance of the two DRL algorithms in a complex MIMO environment, for which the optimal solution is not known. The numerical results confirm the effectiveness of the DRL-based precoding framework and show that the proposed DRL-based framework can outperform the conventional approximation algorithm in the complex MIMO environment.

Figures (10)

• Figure 1: Model-free valued-based and policy-based reinforcement learning algorithms.
• Figure 2: Schematic block diagram of MIMO-OFDM system, where a precoding vector $\hbox{$\bf w$} \in {\mathbb C}^{n_{tx}}$ and a combining vector $\hbox{$\bf r$} \in {\mathbb C}^{n_{rx}}$ are applied, respectively, on per-subband at the transmitter and on per-RE basis at the receiver.
• Figure 3: An illustrative example of time-frequency resource grid of MIMO-OFDM system with resource parameters specified in the 3GPP standards 3gpp:36.211.
• Figure 4: (a) Reinforcement learning through interactions between agent and environment and (b) The interaction between agent and environment can be modeled as Markov-decision process.
• Figure 5: A schematic illustration of the value-based DQN with parameters $\theta$ that can be used for solving the codebook-based precoding problem in the proposed precoding framework, assuming a discrete action space given by ${\mathcal{A}}_d=\left\{a_1,a_2,\cdots,a_N\right\}$.