Rethinking the Spatio-Temporal Alignment of End-to-End 3D Perception

TL;DR

Rethinking spatio-temporal alignment for end-to-end 3D perception, the paper introduces HAT, a plug-and-play module that couples multiple explicit motion models with an adaptive, implicit decoding of motion and semantic cues. By generating diverse anchor and feature hypotheses through a Motion Model Library (MML) and decoding them with a dynamic, query-guided mechanism, HAT delivers robust, adaptive alignment across frames. Across nuScenes and nuScenes-C, HAT yields consistent gains in detection and tracking, achieves state-of-the-art tracking performance when paired with DETR3D, and preserves system stability with modest latency overhead. The results demonstrate that explicit motion modeling, when fused with semantics via learned weighting, remains crucial for reliable end-to-end autonomous driving perception and planning.

Abstract

Spatio-temporal alignment is crucial for temporal modeling of end-to-end (E2E) perception in autonomous driving (AD), providing valuable structural and textural prior information. Existing methods typically rely on the attention mechanism to align objects across frames, simplifying the motion model with a unified explicit physical model (constant velocity, etc.). These approaches prefer semantic features for implicit alignment, challenging the importance of explicit motion modeling in the traditional perception paradigm. However, variations in motion states and object features across categories and frames render this alignment suboptimal. To address this, we propose HAT, a spatio-temporal alignment module that allows each object to adaptively decode the optimal alignment proposal from multiple hypotheses without direct supervision. Specifically, HAT first utilizes multiple explicit motion models to generate spatial anchors and motion-aware feature proposals for historical instances. It then performs multi-hypothesis decoding by incorporating semantic and motion cues embedded in cached object queries, ultimately providing the optimal alignment proposal for the target frame. On nuScenes, HAT consistently improves 3D temporal detectors and trackers across diverse baselines. It achieves state-of-the-art tracking results with 46.0% AMOTA on the test set when paired with the DETR3D detector. In an object-centric E2E AD method, HAT enhances perception accuracy (+1.3% mAP, +3.1% AMOTA) and reduces the collision rate by 32%. When semantics are corrupted (nuScenes-C), the enhancement of motion modeling by HAT enables more robust perception and planning in the E2E AD.

Figures (6)

• Figure 1: The STA module in detection and tracking.
• Figure 2: Architecture of HAT. HAT takes the posterior from the previous frame as input, and outputs the prior structured (anchors) and semantic (features) information for the target frame. We present the alignment process for the $i$-th instance.
• Figure 3: Visualization results of the decoding weights $W_a$ for anchor hypotheses in ADA-Track-HAT on nuScenes. TM means Turning Motion model (CTRV, CTRA), LM means Linear Motion model (CV, CA).
• Figure 4: Visualization results of Sparse4D and Sparse4D-HAT on the BEV plane on nuScenes dataset. The ground-truth, final prediction $B_{t}$ and temporal-aligned detection $B_{t, t-1}$ are depicted as green, blue, and red rectangles.
• Figure 5: The screenshot of our tracking result (ADA-Track ding2024ada-HAT) on the nuScenes test set.