Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

N-DriverMotion: Driver motion learning and prediction using an event-based camera and directly trained spiking neural networks on Loihi 2

Hyo Jong Chung, Byungkon Kang, Yoonseok Yang

TL;DR

A novel optimized four-layer convolutional spiking neural network (CSNN) was trained directly without any time-consuming preprocessing, which enables efficient adaptation to energy- and resource-constrained on-device SNNs for real-time inference on high-resolution event-based streams.

Abstract

Driver motion recognition is a principal factor in ensuring the safety of driving systems. This paper presents a novel system for learning and predicting driver motions and an event-based high-resolution (1280x720) dataset, N-DriverMotion, newly collected to train on a neuromorphic vision system. The system comprises an event-based camera that generates the first high-resolution driver motion dataset representing spike inputs and efficient spiking neural networks (SNNs) that are effective in training and predicting the driver's gestures. The event dataset consists of 13 driver motion categories classified by direction (front, side), illumination (bright, moderate, dark), and participant. A novel simplified four-layer convolutional spiking neural network (CSNN) that we proposed was directly trained using the high-resolution dataset without any time-consuming preprocessing. This enables efficient adaptation to on-device SNNs for real-time inference on high-resolution event-based streams. Compared with recent gesture recognition systems adopting neural networks for vision processing, the proposed neuromorphic vision system achieves comparable accuracy, 94.04\%, in recognizing driver motions with the CSNN architecture. Our proposed CSNN and the dataset can be used to develop safer and more efficient driver monitoring systems for autonomous vehicles or edge devices requiring an efficient neural network architecture.

N-DriverMotion: Driver motion learning and prediction using an event-based camera and directly trained spiking neural networks on Loihi 2

TL;DR

A novel optimized four-layer convolutional spiking neural network (CSNN) was trained directly without any time-consuming preprocessing, which enables efficient adaptation to energy- and resource-constrained on-device SNNs for real-time inference on high-resolution event-based streams.

Abstract

Driver motion recognition is a principal factor in ensuring the safety of driving systems. This paper presents a novel system for learning and predicting driver motions and an event-based high-resolution (1280x720) dataset, N-DriverMotion, newly collected to train on a neuromorphic vision system. The system comprises an event-based camera that generates the first high-resolution driver motion dataset representing spike inputs and efficient spiking neural networks (SNNs) that are effective in training and predicting the driver's gestures. The event dataset consists of 13 driver motion categories classified by direction (front, side), illumination (bright, moderate, dark), and participant. A novel simplified four-layer convolutional spiking neural network (CSNN) that we proposed was directly trained using the high-resolution dataset without any time-consuming preprocessing. This enables efficient adaptation to on-device SNNs for real-time inference on high-resolution event-based streams. Compared with recent gesture recognition systems adopting neural networks for vision processing, the proposed neuromorphic vision system achieves comparable accuracy, 94.04\%, in recognizing driver motions with the CSNN architecture. Our proposed CSNN and the dataset can be used to develop safer and more efficient driver monitoring systems for autonomous vehicles or edge devices requiring an efficient neural network architecture.
Paper Structure (20 sections, 7 equations, 9 figures, 8 tables)

This paper contains 20 sections, 7 equations, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. RELATED WORK
  3. PROPOSED EVENT-BASED DRIVER MOTION RECOGNITION SYSTEM
  4. EXPERIMENTS AND RESULTS
  5. EVENT-BASED CAMERA AND DATA CONVERSION
  6. DATASET
  7. EXPERIMENTAL SETUP AND TOOL FLOW
  8. Multiple driver motion classification
  9. Inference on unseen abnormal driving motions
  10. Energy efficiency in various AI accelerators
  11. NETWORK CONFIGURATION AND TRAINING
  12. EDGE DEVICE CONFIGURATIONS AND INFERENCE
  13. Jetson Xavior NX
  14. Intel Loihi 2
  15. Results
  16. ...and 5 more sections

Figures (9)

  • Figure 1: Simplified structure of Loihi which illustrates how neurocore is structured and operates.
  • Figure 2: The 3D convolutional spiking operation.
  • Figure 3: Proposed Event-based Driver Gesture Recognition System.
  • Figure 4: Sample shots taken for each driver gesture.
  • Figure 5: Sample shots taken of abnormal driver gestures. Note that hand-swipe gestures are not simple hand-waving actions but are actions to protect oneself from various attacks
  • ...and 4 more figures