Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DDD17: End-To-End DAVIS Driving Dataset

Jonathan Binas, Daniel Neil, Shih-Chii Liu, Tobi Delbruck

TL;DR

The paper addresses the need for multimodal, high-temporal-resolution driving data by leveraging event-based DVS alongside standard APS imagery. It introduces DDD17, the first open dataset of annotated DAVIS driving recordings with synchronized APS+DVS data and vehicle telemetry, spanning over 12 hours and 1000+ km under diverse conditions, without LIDAR. It details the data acquisition hardware (DAVIS346B), data formats (HDF5, cAER DAT3.1), and software tools for synchronization, viewing, and export. A preliminary end-to-end steering prediction experiment demonstrates the dataset's feasibility for multimodal driving models and identifies directions for future work and dataset expansion.

Abstract

Event cameras, such as dynamic vision sensors (DVS), and dynamic and active-pixel vision sensors (DAVIS) can supplement other autonomous driving sensors by providing a concurrent stream of standard active pixel sensor (APS) images and DVS temporal contrast events. The APS stream is a sequence of standard grayscale global-shutter image sensor frames. The DVS events represent brightness changes occurring at a particular moment, with a jitter of about a millisecond under most lighting conditions. They have a dynamic range of >120 dB and effective frame rates >1 kHz at data rates comparable to 30 fps (frames/second) image sensors. To overcome some of the limitations of current image acquisition technology, we investigate in this work the use of the combined DVS and APS streams in end-to-end driving applications. The dataset DDD17 accompanying this paper is the first open dataset of annotated DAVIS driving recordings. DDD17 has over 12 h of a 346x260 pixel DAVIS sensor recording highway and city driving in daytime, evening, night, dry and wet weather conditions, along with vehicle speed, GPS position, driver steering, throttle, and brake captured from the car's on-board diagnostics interface. As an example application, we performed a preliminary end-to-end learning study of using a convolutional neural network that is trained to predict the instantaneous steering angle from DVS and APS visual data.

DDD17: End-To-End DAVIS Driving Dataset

TL;DR

The paper addresses the need for multimodal, high-temporal-resolution driving data by leveraging event-based DVS alongside standard APS imagery. It introduces DDD17, the first open dataset of annotated DAVIS driving recordings with synchronized APS+DVS data and vehicle telemetry, spanning over 12 hours and 1000+ km under diverse conditions, without LIDAR. It details the data acquisition hardware (DAVIS346B), data formats (HDF5, cAER DAT3.1), and software tools for synchronization, viewing, and export. A preliminary end-to-end steering prediction experiment demonstrates the dataset's feasibility for multimodal driving models and identifies directions for future work and dataset expansion.

Abstract

Event cameras, such as dynamic vision sensors (DVS), and dynamic and active-pixel vision sensors (DAVIS) can supplement other autonomous driving sensors by providing a concurrent stream of standard active pixel sensor (APS) images and DVS temporal contrast events. The APS stream is a sequence of standard grayscale global-shutter image sensor frames. The DVS events represent brightness changes occurring at a particular moment, with a jitter of about a millisecond under most lighting conditions. They have a dynamic range of >120 dB and effective frame rates >1 kHz at data rates comparable to 30 fps (frames/second) image sensors. To overcome some of the limitations of current image acquisition technology, we investigate in this work the use of the combined DVS and APS streams in end-to-end driving applications. The dataset DDD17 accompanying this paper is the first open dataset of annotated DAVIS driving recordings. DDD17 has over 12 h of a 346x260 pixel DAVIS sensor recording highway and city driving in daytime, evening, night, dry and wet weather conditions, along with vehicle speed, GPS position, driver steering, throttle, and brake captured from the car's on-board diagnostics interface. As an example application, we performed a preliminary end-to-end learning study of using a convolutional neural network that is trained to predict the instantaneous steering angle from DVS and APS visual data.

Paper Structure

This paper contains 8 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Davis Driving Dataset 2017 (DDD17)
  3. DAVIS data
  4. Vehicle control and diagnostic data
  5. Recording and viewing software
  6. Recorded data
  7. Experiments: Steering prediction network
  8. Conclusion

Figures (3)

  • Figure 1: Example scenario visualized by the recording file viewer. The top panels show the DAVIS frames (left; overlaid with some driving data) and events (right), the bottom panel shows a progress bar as well as visualizations of different vehicle data (headlamp status at the top, steering angle in the middle, speed at the bottom).
  • Figure 2: Statistical distribution of various recorded signals.
  • Figure 3: Steering prediction initial result. Comparison of our first APS and DVS steering prediction experiments. A: DVS frame and CNN output. B: APS frame and CNN output. C: segment of time history.