Quantum Deep Reinforcement Learning for Robot Navigation Tasks

TL;DR

It is shown that quantum circuits in hybrid quantum-classic reinforcement learning setups are capable of learning optimal policies in multiple robotic navigation scenarios with notably fewer trainable parameters compared to a classical baseline, and that the classical baseline produces more stable and better performing policies overall.

Abstract

We utilize hybrid quantum deep reinforcement learning to learn navigation tasks for a simple, wheeled robot in simulated environments of increasing complexity. For this, we train parameterized quantum circuits (PQCs) with two different encoding strategies in a hybrid quantum-classical setup as well as a classical neural network baseline with the double deep Q network (DDQN) reinforcement learning algorithm. Quantum deep reinforcement learning (QDRL) has previously been studied in several relatively simple benchmark environments, mainly from the OpenAI gym suite. However, scaling behavior and applicability of QDRL to more demanding tasks closer to real-world problems e. g., from the robotics domain, have not been studied previously. Here, we show that quantum circuits in hybrid quantum-classic reinforcement learning setups are capable of learning optimal policies in multiple robotic navigation scenarios with notably fewer trainable parameters compared to a classical baseline. Across a large number of experimental configurations, we find that the employed quantum circuits outperform the classical neural network baselines when equating for the number of trainable parameters. Yet, the classical neural network consistently showed better results concerning training times and stability, with at least one order of magnitude of trainable parameters more than the best-performing quantum circuits. However, validating the robustness of the learning methods in a large and dynamic environment, we find that the classical baseline produces more stable and better performing policies overall.

Figures (12)

• Figure 1: Main contribution: We use parameterized quantum circuits (PQCs) as function approximators in the DDQN Deep Reinforcement Learning algorithm to learn optimal policies for a simulated Turtlebot robotic system in several simulated navigation tasks.
• Figure 2: Reinforcement Learning setup (left) and main parts of the DDQN algorithm (right). An agent interacts with an environment by performing action $a_{t}$ after observing a state of the environment $\boldsymbol{s}_{t}$, causing a transition to state $\boldsymbol{s}_{t+1}$ and receiving a reward $r_{t}$. For the DDQN algorithm, these interactions are stored in a replay buffer, from which regularly random mini-batches are sampled to train an artificial neural network $Q_{\boldsymbol{\theta}}$ approximating the action-value function.
• Figure 3: Basic principle of a parameterized quantum circuit as function approximator. A unitary $U(\boldsymbol{\theta}, \boldsymbol{x})$, which may be composed of any number of quantum gates, is applied to an $n$ qubit register initialized in its basis state. The unitary is parameterized by trainable parameters $\boldsymbol{\theta}$ and input data $\boldsymbol{x}$. Thereby, the expectation value of an observable $\langle \mathcal{O}\rangle$ can be defined as a parameterized function $f_{\mathcal{O}}(\boldsymbol{\theta}, \boldsymbol{x})$. The parameters $\boldsymbol{\theta}$ are optimized toward a desired outcome, by minimizing a task specific loss $\mathcal{L}$ using a classical optimization technique e. g., gradient descent.
• Figure 4: Data re-upload in parameterized quantum circuits: After an initial parameterized unitary $V(\boldsymbol{\theta}_0)$, $L$ layers of data encoding unitaries $U_\text{in}(\boldsymbol{x}_l)$ and parameterized unitaries $V(\boldsymbol{\theta}_l)$ are applied to the $n$ qubit quantum register.
• Figure 5: The three simulated static navigation environments for the Turtlebot 2 robot. In each, the robot has to navigate from its starting position in the upper left corner to the position marked with a green circle in the lower right while avoiding collisions with the enclosing walls and any obstacles. With the configured control scheme, this takes about 20 steps in the 3$\times$3 environment (left), 30 in the 4$\times$4 (center), and 45 steps in the 5$\times$5 environment (right) for a (near) optimal trajectory. Possible paths the robot can take to solve each environment are marked with a red dotted line.