An Integrated Approach to Neural Architecture Search for Deep Q-Networks

TL;DR

The paper addresses the fixed-architecture limitation in deep reinforcement learning by introducing NAS-DQN, which embeds a lightweight, performance-guided NAS controller into the DRL training loop to adapt the Q-network online. The approach defines an architecture space with $L \in \{2,3,4\}$ layers, $U \in \{32,64,128\}$ units per layer, and activations in $\{\text{ReLU}, \text{tanh}, \text{Leaky ReLU}\}$, and uses a softmax-based selection biased by past performance with temperature $\tau=1.5$ and decay $\rho_{\epsilon}=0.95$. Implemented on a custom inverted-pendulum task with a Double DQN backbone, NAS-DQN outperforms fixed architectures and Random-NAS in final performance, sample efficiency, and policy stability, while incurring negligible extra computational cost. The findings demonstrate that online, intelligent architecture adaptation is a practical and effective component of RL, enabling a dynamic co-evolution of computation and learning rather than a static offline design.

Abstract

The performance of deep reinforcement learning agents is fundamentally constrained by their neural network architecture, a choice traditionally made through expensive hyperparameter searches and then fixed throughout training. This work investigates whether online, adaptive architecture optimization can escape this constraint and outperform static designs. We introduce NAS-DQN, an agent that integrates a learned neural architecture search controller directly into the DRL training loop, enabling dynamic network reconfiguration based on cumulative performance feedback. We evaluate NAS-DQN against three fixed-architecture baselines and a random search control on a continuous control task, conducting experiments over multiple random seeds. Our results demonstrate that NAS-DQN achieves superior final performance, sample efficiency, and policy stability while incurring negligible computational overhead. Critically, the learned search strategy substantially outperforms both undirected random architecture exploration and poorly-chosen fixed designs, indicating that intelligent, performance-guided search is the key mechanism driving success. These findings establish that architecture adaptation is not merely beneficial but necessary for optimal sample efficiency in online deep reinforcement learning, and suggest that the design of RL agents need not be a static offline choice but can instead be seamlessly integrated as a dynamic component of the learning process itself.

Figures (6)

• Figure 1: Visualizations of the three fixed Q-Network architectures used as baselines. Each network takes the 3-dimensional state vector as input and produces Q-values for the 3 discrete actions.
• Figure 2: NAS-DQN achieves faster convergence and higher mean performance with lower variance across all three seeds.
• Figure 3: NAS-DQN converges approximately 3× faster than fixed baselines and 10× faster than Random-NAS.
• Figure 4: NAS-DQN incurs negligible overhead compared to fixed agents despite delivering superior performance.
• Figure 5: NAS-DQN concentrates search in high-reward regions; Random-NAS explores uniformly across the space.