Interactive Double Deep Q-network: Integrating Human Interventions and Evaluative Predictions in Reinforcement Learning of Autonomous Driving

TL;DR

This work presents Interactive Double Deep Q-Network (iDDQN), a human-in-the-loop reinforcement learning framework for autonomous driving that blends real-time human interventions with agent decisions through a decaying weight $\lambda_h$. By extending Clipped Double Q-Learning, iDDQN computes a composite Q-value $Q_{combined}$ that favors human actions early in training and gradually shifts toward autonomous policy, with a robust target $Q_{target}$ to mitigate overestimation. An offline Evaluation Prediction Module (EPM) complements the approach by predicting counterfactual outcomes and assessing alignment between human and agent actions, reporting a high agreement rate when human input leads to better rewards. Experimental results in AirSim demonstrate that iDDQN with a decayed human weight outperforms BC, HG-DAgger, DQfD, and vanilla DRL, with strong generalization to unseen environments and quantifiable benefits from human guidance. The work contributes a practical HITL framework, an evaluative mechanism for human interventions, and insights into how to schedule human influence for efficient, safer autonomous driving policies.

Abstract

Integrating human expertise with machine learning is crucial for applications demanding high accuracy and safety, such as autonomous driving. This study introduces Interactive Double Deep Q-network (iDDQN), a Human-in-the-Loop (HITL) approach that enhances Reinforcement Learning (RL) by merging human insights directly into the RL training process, improving model performance. Our proposed iDDQN method modifies the Q-value update equation to integrate human and agent actions, establishing a collaborative approach for policy development. Additionally, we present an offline evaluative framework that simulates the agent's trajectory as if no human intervention had occurred, to assess the effectiveness of human interventions. Empirical results in simulated autonomous driving scenarios demonstrate that iDDQN outperforms established approaches, including Behavioral Cloning (BC), HG-DAgger, Deep Q-Learning from Demonstrations (DQfD), and vanilla DRL in leveraging human expertise for improving performance and adaptability.

Figures (5)

• Figure 1: AirSim environments utilized for model training and evaluation. Left: residential training environment; right: coastal testing environment.
• Figure 2: Performance comparison of iDDQN with varying $\lambda_h$. The $\lambda_h = \text{decay}$ configuration achieved the best results, balancing early human guidance and gradual autonomy.
• Figure 3: Comparison of episodic rewards during training for iDDQN and baseline methods. BC is trained for 15K steps (expert demonstrations) and HG-DAgger for 21K steps in total (after the 4th iteration). While BC and HG-DAgger struggled with distributional shift, DQfD leveraged pretraining effectively but was outperformed by iDDQN with the ($\lambda_h = \text{decay}$).
• Figure 4: The top row depicts the actual trajectory executed when a human intervened. In contrast, the bottom row shows the predicted trajectory generated by the EPM if the agent had acted autonomously without intervention. The cumulative reward achieved with human intervention ($\sum r_{\text{actual}}=2.22$) is higher than the reward predicted for the agent's decision ($\sum r_{\text{agent}}=2.04$), indicating that the EPM aligns with the human intervention.
• Figure 5: Ablation studies for key hyperparameters. (a) Examines the effect of varying $\delta$ and $\beta$ on reward stabilization, showing a slight performance improvement at $\delta=0.2$ and $\beta=0.2$. (b) Analyzes $\xi$, demonstrating better reward at $\xi=0.5$ compared to other values. (c) Investigates the impact of doubling human intervention frequency for $\lambda_h=1$, showing better early performance, but at the cost of doubling the intervention limit. This suggests that increasing human interventions does not necessarily result in proportional performance gains, highlighting the need for optimized intervention scheduling.