Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Probabilistic 3D Multi-Object Cooperative Tracking for Autonomous Driving via Differentiable Multi-Sensor Kalman Filter

Hsu-kuang Chiu, Chien-Yi Wang, Min-Hung Chen, Stephen F. Smith

TL;DR

This work tackles the reliability gaps in autonomous driving perception under occlusion by enabling 3D multi-object cooperative tracking via V2V communication. It introduces DMSTrack, a differentiable multi-sensor Kalman Filter that learns per-detection observation covariances using a covariance neural network fed by local BEV and positional features, enabling better state estimation across multiple vehicles. Evaluated on the V2V4Real dataset, DMSTrack achieves a 17% AMOTA improvement with only 0.037x the communication cost of the prior CoBEVT approach, highlighting both accuracy and efficiency gains. The approach advances cooperative tracking by integrating probabilistic state estimation with learned sensor uncertainty, offering practical benefits for robust multi-vehicle perception in real-world driving scenarios.

Abstract

Current state-of-the-art autonomous driving vehicles mainly rely on each individual sensor system to perform perception tasks. Such a framework's reliability could be limited by occlusion or sensor failure. To address this issue, more recent research proposes using vehicle-to-vehicle (V2V) communication to share perception information with others. However, most relevant works focus only on cooperative detection and leave cooperative tracking an underexplored research field. A few recent datasets, such as V2V4Real, provide 3D multi-object cooperative tracking benchmarks. However, their proposed methods mainly use cooperative detection results as input to a standard single-sensor Kalman Filter-based tracking algorithm. In their approach, the measurement uncertainty of different sensors from different connected autonomous vehicles (CAVs) may not be properly estimated to utilize the theoretical optimality property of Kalman Filter-based tracking algorithms. In this paper, we propose a novel 3D multi-object cooperative tracking algorithm for autonomous driving via a differentiable multi-sensor Kalman Filter. Our algorithm learns to estimate measurement uncertainty for each detection that can better utilize the theoretical property of Kalman Filter-based tracking methods. The experiment results show that our algorithm improves the tracking accuracy by 17% with only 0.037x communication costs compared with the state-of-the-art method in V2V4Real. Our code and videos are available at https://github.com/eddyhkchiu/DMSTrack/ and https://eddyhkchiu.github.io/dmstrack.github.io/ .

Probabilistic 3D Multi-Object Cooperative Tracking for Autonomous Driving via Differentiable Multi-Sensor Kalman Filter

TL;DR

This work tackles the reliability gaps in autonomous driving perception under occlusion by enabling 3D multi-object cooperative tracking via V2V communication. It introduces DMSTrack, a differentiable multi-sensor Kalman Filter that learns per-detection observation covariances using a covariance neural network fed by local BEV and positional features, enabling better state estimation across multiple vehicles. Evaluated on the V2V4Real dataset, DMSTrack achieves a 17% AMOTA improvement with only 0.037x the communication cost of the prior CoBEVT approach, highlighting both accuracy and efficiency gains. The approach advances cooperative tracking by integrating probabilistic state estimation with learned sensor uncertainty, offering practical benefits for robust multi-vehicle perception in real-world driving scenarios.

Abstract

Current state-of-the-art autonomous driving vehicles mainly rely on each individual sensor system to perform perception tasks. Such a framework's reliability could be limited by occlusion or sensor failure. To address this issue, more recent research proposes using vehicle-to-vehicle (V2V) communication to share perception information with others. However, most relevant works focus only on cooperative detection and leave cooperative tracking an underexplored research field. A few recent datasets, such as V2V4Real, provide 3D multi-object cooperative tracking benchmarks. However, their proposed methods mainly use cooperative detection results as input to a standard single-sensor Kalman Filter-based tracking algorithm. In their approach, the measurement uncertainty of different sensors from different connected autonomous vehicles (CAVs) may not be properly estimated to utilize the theoretical optimality property of Kalman Filter-based tracking algorithms. In this paper, we propose a novel 3D multi-object cooperative tracking algorithm for autonomous driving via a differentiable multi-sensor Kalman Filter. Our algorithm learns to estimate measurement uncertainty for each detection that can better utilize the theoretical property of Kalman Filter-based tracking methods. The experiment results show that our algorithm improves the tracking accuracy by 17% with only 0.037x communication costs compared with the state-of-the-art method in V2V4Real. Our code and videos are available at https://github.com/eddyhkchiu/DMSTrack/ and https://eddyhkchiu.github.io/dmstrack.github.io/ .
Paper Structure (23 sections, 10 equations, 4 figures, 2 tables)

This paper contains 23 sections, 10 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Single-Agent Autonomous Driving
  4. Cooperative Perception with CAVs
  5. Kalman Filters
  6. Method
  7. Kalman Filter Setting
  8. Object State
  9. Prediction Step
  10. Data Association and Update Step
  11. Input Feature Formation
  12. Local BEV Feature
  13. Positional Feature
  14. Covariance Neural Network
  15. Loss Function and Training Strategy
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Cooperative tracking architecture comparison.
  • Figure 2: Architecture diagram of our proposed cooperative tracking algorithm applied to a minimal example with two CAVs.
  • Figure 3: Covariance Neural Network. The numbers represent features' tensor shapes and trainable layers' input, output, and kernel sizes.
  • Figure 4: Qualitative tracking results of our algorithm.