Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

End-To-End Training and Testing Gamification Framework to Learn Human Highway Driving

Satya R. Jaladi, Zhimin Chen, Narahari R. Malayanur, Raja M. Macherla, Bing Li

TL;DR

This work presents a game-based end-to-end learning framework for highway driving that learns from human behavior in Grand Theft Auto V. It compares two architectures, a NVIDIA end-to-end network and a VGG-19-based model, using transfer learning for the latter and training both on a large GTA V dataset to predict steering and throttle from in-game images. Results show that VGG-19 yields better generalization and faster convergence, though real-time performance is limited by hardware to a few frames per second. The study demonstrates the viability of end-to-end, game-based learning for autonomous highway driving and provides a dataset and framework for future research, while highlighting the need for improved inference speed and human-subject evaluation.

Abstract

The current autonomous stack is well modularized and consists of perception, decision making and control in a handcrafted framework. With the advances in artificial intelligence (AI) and computing resources, researchers have been pushing the development of end-to-end AI for autonomous driving, at least in problems of small searching space such as in highway scenarios, and more and more photorealistic simulation will be critical for efficient learning. In this research, we propose a novel game-based end-to-end learning and testing framework for autonomous vehicle highway driving, by learning from human driving skills. Firstly, we utilize the popular game Grand Theft Auto V (GTA V) to collect highway driving data with our proposed programmable labels. Then, an end-to-end architecture predicts the steering and throttle values that control the vehicle by the image of the game screen. The predicted control values are sent to the game via a virtual controller to keep the vehicle in lane and avoid collisions with other vehicles on the road. The proposed solution is validated in GTA V games, and the results demonstrate the effectiveness of this end-to-end gamification framework for learning human driving skills.

End-To-End Training and Testing Gamification Framework to Learn Human Highway Driving

TL;DR

This work presents a game-based end-to-end learning framework for highway driving that learns from human behavior in Grand Theft Auto V. It compares two architectures, a NVIDIA end-to-end network and a VGG-19-based model, using transfer learning for the latter and training both on a large GTA V dataset to predict steering and throttle from in-game images. Results show that VGG-19 yields better generalization and faster convergence, though real-time performance is limited by hardware to a few frames per second. The study demonstrates the viability of end-to-end, game-based learning for autonomous highway driving and provides a dataset and framework for future research, while highlighting the need for improved inference speed and human-subject evaluation.

Abstract

The current autonomous stack is well modularized and consists of perception, decision making and control in a handcrafted framework. With the advances in artificial intelligence (AI) and computing resources, researchers have been pushing the development of end-to-end AI for autonomous driving, at least in problems of small searching space such as in highway scenarios, and more and more photorealistic simulation will be critical for efficient learning. In this research, we propose a novel game-based end-to-end learning and testing framework for autonomous vehicle highway driving, by learning from human driving skills. Firstly, we utilize the popular game Grand Theft Auto V (GTA V) to collect highway driving data with our proposed programmable labels. Then, an end-to-end architecture predicts the steering and throttle values that control the vehicle by the image of the game screen. The predicted control values are sent to the game via a virtual controller to keep the vehicle in lane and avoid collisions with other vehicles on the road. The proposed solution is validated in GTA V games, and the results demonstrate the effectiveness of this end-to-end gamification framework for learning human driving skills.
Paper Structure (10 sections, 9 figures)

This paper contains 10 sections, 9 figures.

Table of Contents

  1. Introduction
  2. Related works
  3. End-To-End Learning Framework
  4. Experiments
  5. Data Collection Strategies
  6. Image Pre-Processing and Data Augmentation
  7. Training the Networks With Collected Data
  8. Model Inference
  9. Results and Discussion
  10. Conclusion

Figures (9)

  • Figure 1: Framework methodology explaining the methodology implemented to perform data collection, training and inference
  • Figure 2: Data collection system (While the user plays the game, the game footage and control values are simultaneously collected)
  • Figure 3: Virtual controller software (Used to transfer output from neural network to the in-game vehicle)
  • Figure 4: Image cropping (Only a certain section of the screen was fed into the algorithm to avoid redundant data)
  • Figure 5: Image left right flipping (Augmenting data by flipping images)
  • ...and 4 more figures