Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-task Learning for Real-time Autonomous Driving Leveraging Task-adaptive Attention Generator

Wonhyeok Choi, Mingyu Shin, Hyukzae Lee, Jaehoon Cho, Jaehyeon Park, Sunghoon Im

TL;DR

The paper tackles real-time autonomous driving perception across three tasks—monocular 3D object detection, semantic segmentation, and dense depth estimation—addressing negative transfer in heterogeneous multi-task learning. It introduces a hard-shared, two-pathway backbone augmented by a Task-adaptive Attention Generator (TAG) and a compact multi-head design to enable efficient joint learning. TAG leverages a per-task channel attention from the semantic branch and a shared spatial attention from the detail branch to produce task-specific features without sacrificing speed. Experiments on Cityscapes-3D show state-of-the-art performance with favorable Delta_T improvements and notable inference-time reductions, supported by comprehensive ablations. Overall, the approach advances practical, real-time multi-task perception for autonomous driving by balancing accuracy and efficiency across tasks.

Abstract

Real-time processing is crucial in autonomous driving systems due to the imperative of instantaneous decision-making and rapid response. In real-world scenarios, autonomous vehicles are continuously tasked with interpreting their surroundings, analyzing intricate sensor data, and making decisions within split seconds to ensure safety through numerous computer vision tasks. In this paper, we present a new real-time multi-task network adept at three vital autonomous driving tasks: monocular 3D object detection, semantic segmentation, and dense depth estimation. To counter the challenge of negative transfer, which is the prevalent issue in multi-task learning, we introduce a task-adaptive attention generator. This generator is designed to automatically discern interrelations across the three tasks and arrange the task-sharing pattern, all while leveraging the efficiency of the hard-parameter sharing approach. To the best of our knowledge, the proposed model is pioneering in its capability to concurrently handle multiple tasks, notably 3D object detection, while maintaining real-time processing speeds. Our rigorously optimized network, when tested on the Cityscapes-3D datasets, consistently outperforms various baseline models. Moreover, an in-depth ablation study substantiates the efficacy of the methodologies integrated into our framework.

Multi-task Learning for Real-time Autonomous Driving Leveraging Task-adaptive Attention Generator

TL;DR

The paper tackles real-time autonomous driving perception across three tasks—monocular 3D object detection, semantic segmentation, and dense depth estimation—addressing negative transfer in heterogeneous multi-task learning. It introduces a hard-shared, two-pathway backbone augmented by a Task-adaptive Attention Generator (TAG) and a compact multi-head design to enable efficient joint learning. TAG leverages a per-task channel attention from the semantic branch and a shared spatial attention from the detail branch to produce task-specific features without sacrificing speed. Experiments on Cityscapes-3D show state-of-the-art performance with favorable Delta_T improvements and notable inference-time reductions, supported by comprehensive ablations. Overall, the approach advances practical, real-time multi-task perception for autonomous driving by balancing accuracy and efficiency across tasks.

Abstract

Real-time processing is crucial in autonomous driving systems due to the imperative of instantaneous decision-making and rapid response. In real-world scenarios, autonomous vehicles are continuously tasked with interpreting their surroundings, analyzing intricate sensor data, and making decisions within split seconds to ensure safety through numerous computer vision tasks. In this paper, we present a new real-time multi-task network adept at three vital autonomous driving tasks: monocular 3D object detection, semantic segmentation, and dense depth estimation. To counter the challenge of negative transfer, which is the prevalent issue in multi-task learning, we introduce a task-adaptive attention generator. This generator is designed to automatically discern interrelations across the three tasks and arrange the task-sharing pattern, all while leveraging the efficiency of the hard-parameter sharing approach. To the best of our knowledge, the proposed model is pioneering in its capability to concurrently handle multiple tasks, notably 3D object detection, while maintaining real-time processing speeds. Our rigorously optimized network, when tested on the Cityscapes-3D datasets, consistently outperforms various baseline models. Moreover, an in-depth ablation study substantiates the efficacy of the methodologies integrated into our framework.
Paper Structure (21 sections, 6 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 21 sections, 6 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Monocular 3D object detection
  4. Multi-Task Learning (MTL)
  5. Real-time semantic segmentation
  6. Method
  7. Enhanced two-pathway Architecture
  8. Task-adaptive attention generator (TAG)
  9. Multi-head Structure
  10. Loss function
  11. Experiments
  12. Dataset
  13. Cityscapes-3D
  14. Baselines
  15. Evaluation Metric
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Speed-performance trade-off curve for the Cityscapes-3D validation set (the top-right quadrant is optimal). The Y-axis represents the average relative performance compared to the single-task setup.
  • Figure 2: Comparison between existing backbones and our backbone.
  • Figure 3: The overall pipeline of the proposed network.
  • Figure 4: Feature heatmap visualization with respect to attention in TAG module.