Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhanced Deep Q-Learning for 2D Self-Driving Cars: Implementation and Evaluation on a Custom Track Environment

Sagar Pathak, Bidhya Shrestha, Kritish Pahi

TL;DR

The paper tackles the challenge of training an autonomous driving agent in a risk-free setting by implementing a 2D self-driving car in a Pygame-based environment with seven forward-looking sensors and a three-action space. It advances DQN-based control by introducing a priority-based action selection mechanism and demonstrates that this variant significantly improves average rewards and training efficiency compared with a standard DQN and a vanilla neural network. Key contributions include a complete DQN implementation, a sensor-driven observation space, and a comparative evaluation highlighting the importance of action-selection strategies in RL-driven driving tasks. The work provides a compact, accessible testbed for RL in autonomous driving and outlines concrete future steps, such as integrating SUMO for multi-vehicle dynamics and exploring more sophisticated reward structures to enhance realism and generalization.

Abstract

This research project presents the implementation of a Deep Q-Learning Network (DQN) for a self-driving car on a 2-dimensional (2D) custom track, with the objective of enhancing the DQN network's performance. It encompasses the development of a custom driving environment using Pygame on a track surrounding the University of Memphis map, as well as the design and implementation of the DQN model. The algorithm utilizes data from 7 sensors installed in the car, which measure the distance between the car and the track. These sensors are positioned in front of the vehicle, spaced 20 degrees apart, enabling them to sense a wide area ahead. We successfully implemented the DQN and also a modified version of the DQN with a priority-based action selection mechanism, which we refer to as modified DQN. The model was trained over 1000 episodes, and the average reward received by the agent was found to be around 40, which is approximately 60% higher than the original DQN and around 50% higher than the vanilla neural network.

Enhanced Deep Q-Learning for 2D Self-Driving Cars: Implementation and Evaluation on a Custom Track Environment

TL;DR

The paper tackles the challenge of training an autonomous driving agent in a risk-free setting by implementing a 2D self-driving car in a Pygame-based environment with seven forward-looking sensors and a three-action space. It advances DQN-based control by introducing a priority-based action selection mechanism and demonstrates that this variant significantly improves average rewards and training efficiency compared with a standard DQN and a vanilla neural network. Key contributions include a complete DQN implementation, a sensor-driven observation space, and a comparative evaluation highlighting the importance of action-selection strategies in RL-driven driving tasks. The work provides a compact, accessible testbed for RL in autonomous driving and outlines concrete future steps, such as integrating SUMO for multi-vehicle dynamics and exploring more sophisticated reward structures to enhance realism and generalization.

Abstract

This research project presents the implementation of a Deep Q-Learning Network (DQN) for a self-driving car on a 2-dimensional (2D) custom track, with the objective of enhancing the DQN network's performance. It encompasses the development of a custom driving environment using Pygame on a track surrounding the University of Memphis map, as well as the design and implementation of the DQN model. The algorithm utilizes data from 7 sensors installed in the car, which measure the distance between the car and the track. These sensors are positioned in front of the vehicle, spaced 20 degrees apart, enabling them to sense a wide area ahead. We successfully implemented the DQN and also a modified version of the DQN with a priority-based action selection mechanism, which we refer to as modified DQN. The model was trained over 1000 episodes, and the average reward received by the agent was found to be around 40, which is approximately 60% higher than the original DQN and around 50% higher than the vanilla neural network.
Paper Structure (15 sections, 9 figures, 3 tables, 3 algorithms)

This paper contains 15 sections, 9 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Reinforcement Learning
  3. Deep Q-Learning Network (DQN)
  4. PyGame Learning Environment
  5. Motivation
  6. Related Work
  7. Reinforcement Learning (RL) Environment
  8. SUMO Framework for Simulation
  9. Methodology
  10. DQN Implementation
  11. Priority Algorithm
  12. Experiments and Results
  13. Discussion
  14. Conclusion
  15. Future Works

Figures (9)

  • Figure 1: Flowchart of reinforcement learning
  • Figure 2: Proposed Deep RL Framework for autonomous driving by Ahmad El Sallab et. al.
  • Figure 3: Double DQN structure proposed by Max Peter et. al.
  • Figure 4: Selected region for the vehicle simulation around the University of Memphis in OpenStreetMap
  • Figure 5: Network Map after converted using SUMO netconvert tool
  • ...and 4 more figures