QADQN: Quantum Attention Deep Q-Network for Financial Market Prediction

TL;DR

The QADQN architecture uses a variational quantum circuit inside a traditional deep Q-learning framework to take advantage of possible quantum advantages in decision-making, and evaluates the agent's learning process by examining its reward accumulation and the effectiveness of its experience replay mechanism.

Abstract

Financial market prediction and optimal trading strategy development remain challenging due to market complexity and volatility. Our research in quantum finance and reinforcement learning for decision-making demonstrates the approach of quantum-classical hybrid algorithms to tackling real-world financial challenges. In this respect, we corroborate the concept with rigorous backtesting and validate the framework's performance under realistic market conditions, by including fixed transaction cost per trade. This paper introduces a Quantum Attention Deep Q-Network (QADQN) approach to address these challenges through quantum-enhanced reinforcement learning. Our QADQN architecture uses a variational quantum circuit inside a traditional deep Q-learning framework to take advantage of possible quantum advantages in decision-making. We gauge the QADQN agent's performance on historical data from major market indices, including the S&P 500. We evaluate the agent's learning process by examining its reward accumulation and the effectiveness of its experience replay mechanism. Our empirical results demonstrate the QADQN's superior performance, achieving better risk-adjusted returns with Sortino ratios of 1.28 and 1.19 for non-overlapping and overlapping test periods respectively, indicating effective downside risk management.

Figures (3)

• Figure 1: QADQN Agent Architecture: The architecture begins with an LSTM network, which processes the input features along with the previous hidden and cell states $(h(t-1), c(t-1))$ to produce the current hidden and cell states $(h(t), c(t))$. The output of the LSTM is then passed through a pre-net consisting of linear layers reducing the dimensionality from 64 to 32, then to 8, and finally to 4, which is the number of qubits. These outputs are fed into quantum layers to generate the Queries (Q), Keys (K), and Values (V) for the scaled dot-product attention mechanism. The attention mechanism integrates these quantum-derived components, followed by concatenation and an additional quantum layer to output Q-values, which are used for action selection in OHLC data processing.
• Figure 2: Quantum layer circuit: It begins with data encoding using angle embedding, where input data is encoded into the quantum state via parameterized rotation gates $R_X(\theta_i)$. The encoded quantum states are then processed by a parameterized quantum circuit consisting of additional parameterized rotation gates and Controlled-NOT gates, facilitating entanglement and complex quantum state manipulation. The quantum states are subsequently measured to obtain classical outputs. During training, the classical outputs compute the Huber loss, and the parameters $\theta_i$ are updated using the Lion optimizer.
• Figure 3: QADQN Agent trading simulation for S&P 500 showcasing trading performance over time.