Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NeurAll: Towards a Unified Visual Perception Model for Automated Driving

Ganesh Sistu, Isabelle Leang, Sumanth Chennupati, Senthil Yogamani, Ciaran Hughes, Stefan Milz, Samir Rawashdeh

TL;DR

NeurAll tackles unified perception for automated driving by sharing early CNN layers across tasks to reduce computation and improve generalization. It introduces a shared encoder with task-specific decoders and applies techniques such as task loss weighting, multi-stream inputs, auxiliary learning, and synergized decoders to enable scalable multi-task learning, including a two-stream demonstration. Empirical results show temporal multi-streams yield IoU gains for segmentation and depth, auxiliary depth learning provides IoU improvements of about $4$% on KITTI and $3$% on SYNTHIA, and a three-task model using the product of losses achieves competitive performance with a compact parameter budget. The work also discusses deployment considerations on embedded hardware (e.g., approximately $30$ TOPS) and outlines a path toward larger, multi-task datasets to realize a fully unified perception stack for automated driving.

Abstract

Convolutional Neural Networks (CNNs) are successfully used for the important automotive visual perception tasks including object recognition, motion and depth estimation, visual SLAM, etc. However, these tasks are typically independently explored and modeled. In this paper, we propose a joint multi-task network design for learning several tasks simultaneously. Our main motivation is the computational efficiency achieved by sharing the expensive initial convolutional layers between all tasks. Indeed, the main bottleneck in automated driving systems is the limited processing power available on deployment hardware. There is also some evidence for other benefits in improving accuracy for some tasks and easing development effort. It also offers scalability to add more tasks leveraging existing features and achieving better generalization. We survey various CNN based solutions for visual perception tasks in automated driving. Then we propose a unified CNN model for the important tasks and discuss several advanced optimization and architecture design techniques to improve the baseline model. The paper is partly review and partly positional with demonstration of several preliminary results promising for future research. We first demonstrate results of multi-stream learning and auxiliary learning which are important ingredients to scale to a large multi-task model. Finally, we implement a two-stream three-task network which performs better in many cases compared to their corresponding single-task models, while maintaining network size.

NeurAll: Towards a Unified Visual Perception Model for Automated Driving

TL;DR

NeurAll tackles unified perception for automated driving by sharing early CNN layers across tasks to reduce computation and improve generalization. It introduces a shared encoder with task-specific decoders and applies techniques such as task loss weighting, multi-stream inputs, auxiliary learning, and synergized decoders to enable scalable multi-task learning, including a two-stream demonstration. Empirical results show temporal multi-streams yield IoU gains for segmentation and depth, auxiliary depth learning provides IoU improvements of about % on KITTI and % on SYNTHIA, and a three-task model using the product of losses achieves competitive performance with a compact parameter budget. The work also discusses deployment considerations on embedded hardware (e.g., approximately TOPS) and outlines a path toward larger, multi-task datasets to realize a fully unified perception stack for automated driving.

Abstract

Convolutional Neural Networks (CNNs) are successfully used for the important automotive visual perception tasks including object recognition, motion and depth estimation, visual SLAM, etc. However, these tasks are typically independently explored and modeled. In this paper, we propose a joint multi-task network design for learning several tasks simultaneously. Our main motivation is the computational efficiency achieved by sharing the expensive initial convolutional layers between all tasks. Indeed, the main bottleneck in automated driving systems is the limited processing power available on deployment hardware. There is also some evidence for other benefits in improving accuracy for some tasks and easing development effort. It also offers scalability to add more tasks leveraging existing features and achieving better generalization. We survey various CNN based solutions for visual perception tasks in automated driving. Then we propose a unified CNN model for the important tasks and discuss several advanced optimization and architecture design techniques to improve the baseline model. The paper is partly review and partly positional with demonstration of several preliminary results promising for future research. We first demonstrate results of multi-stream learning and auxiliary learning which are important ingredients to scale to a large multi-task model. Finally, we implement a two-stream three-task network which performs better in many cases compared to their corresponding single-task models, while maintaining network size.

Paper Structure

This paper contains 25 sections, 1 equation, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Automotive Visual Perception Tasks
  3. Object recognition
  4. Generic object detection
  5. Odometry, Localization and Mapping
  6. Other tasks
  7. Unified visual perception model
  8. Motivation
  9. Universality of CNN features
  10. Adaptation of unified model
  11. Hardware trend
  12. Pros and cons of unified model
  13. Pros
  14. Cons
  15. Proposed baseline of unified model
  16. ...and 10 more sections

Figures (3)

  • Figure 1: NeurAll: Proposed unified CNN architecture for the important visual perception tasks in automated driving
  • Figure 2: Two-stream three-task visual perception network architecture performing segmentation, depth estimation and moving object detection
  • Figure 3: Qualitative results of two-stream three-task model performing segmentation, depth estimation and motion segmentation. Left to Right: Input Image, Single Task Network outputs, Multitask Output, Ground Truth.