S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking

TL;DR

This work targets robust end-to-end 3D multi-object tracking for autonomous driving, addressing challenges posed by occlusions and small objects. It decomposes the tracking pipeline into query initialization, propagation, and matching, and introduces three modules: 2D-Prompted Query Initialization to bootstrap accurate 3D query locations, Uncertainty-aware Probabilistic Decoder to model predictive uncertainty with Gaussian attention, and Hierarchical Query Denoising to improve training robustness and convergence. Empirical results on nuScenes show state-of-the-art performance, with 66.3% AMOTA on the test set and a substantial margin over prior end-to-end trackers, along with strong ablations validating each module’s contribution. The proposed S2-Track thus offers a simple yet powerful framework that enhances end-to-end 3D MOT and is well-positioned for future sensor fusion extensions and real-world deployment.

Abstract

3D multiple object tracking (MOT) plays a crucial role in autonomous driving perception. Recent end-to-end query-based trackers simultaneously detect and track objects, which have shown promising potential for the 3D MOT task. However, existing methods are still in the early stages of development and lack systematic improvements, failing to track objects in certain complex scenarios, like occlusions and the small size of target object's situations. In this paper, we first summarize the current end-to-end 3D MOT framework by decomposing it into three constituent parts: query initialization, query propagation, and query matching. Then we propose corresponding improvements, which lead to a strong yet simple tracker: S2-Track. Specifically, for query initialization, we present 2D-Prompted Query Initialization, which leverages predicted 2D object and depth information to prompt an initial estimate of the object's 3D location. For query propagation, we introduce an Uncertainty-aware Probabilistic Decoder to capture the uncertainty of complex environment in object prediction with probabilistic attention. For query matching, we propose a Hierarchical Query Denoising strategy to enhance training robustness and convergence. As a result, our S2-Track achieves state-of-the-art performance on nuScenes benchmark, i.e., 66.3% AMOTA on test split, surpassing the previous best end-to-end solution by a significant margin of 8.9% AMOTA. We achieve 1st place on the nuScenes tracking task leaderboard.

Figures (7)

• Figure 1: (a) In complex driving scenarios, there are various challenge factors, e.g., the occlusions and small size of target objects, which present significant challenges to achieving accurate tracking. The previous state-of-the-art end-to-end tracker, PF-Track pang2023PFtrack, fails to track objects in certain complex scenarios. (b) Our S2-Track proposes three simple yet strong modules to enhance baseline comprehensively, leading to improved tracking performance.
• Figure 2: S2-Track framework. The proposed 2D-Prompted Query Initialization (PQI), Uncertainty-aware Probabilistic Decoder (UPD), and Hierarchical Query Denoising (HQD) are incorporated together to improve tracking performance. Neg: negative, Ign: ignore, Pos: positive, Mask: separate the normal queries and the denoising part to prevent information leakage.
• Figure 3: Uncertainty-aware Probabilistic Decoder (UPD) architecture. The traditional cross-attention is upgraded with probabilistic attention to quantifying the uncertainty. The probabilistic attention utilizes a multi-layer perception that takes the query $q$ and key $k$ as input to generate the mean and standard deviation, which are used to form a Gaussian distribution. Subsequently, the attention value $\alpha$ is sampled from the constructed Gaussian distribution.
• Figure 4: Qualitative results on the nuScenes dataset. (a) The tracking results for an occlusion scenario of two pedestrians of consecutive frames ($t_i-t_{i+12}$). (b) The tracking results on several challenging tracking scenes. Moreover, we plot the attention scores of object queries, which indicate how strongly the model focuses on the target objects. A higher concentration of color represents a higher attention score and a stronger confidence in the corresponding object.
• Figure 5: Details of the depth network in PQI.