TrackTeller: Temporal Multimodal 3D Grounding for Behavior-Dependent Object References

TL;DR

TrackTeller tackles temporal language-based 3D grounding in dynamic driving scenes by fusing LiDAR and multi-view imagery into a unified UniScene representation and aligning it with natural language. It introduces a two-stage language-conditioned decoding pipeline (LSM and LGD) and augments grounding with a memory- and future-aware temporal reasoning module to capture short-term motion. The full framework is trained with a multi-task objective that jointly optimizes detection, memory consistency, future forecasting, and grounding accuracy. On the NuPrompt benchmark, TrackTeller delivers state-of-the-art performance, significantly improving AMOTA and reducing false alarms, while maintaining real-time inference, enabling robust, language-driven perception for autonomous driving in motion-rich environments.

Abstract

Understanding natural-language references to objects in dynamic 3D driving scenes is essential for interactive autonomous systems. In practice, many referring expressions describe targets through recent motion or short-term interactions, which cannot be resolved from static appearance or geometry alone. We study temporal language-based 3D grounding, where the objective is to identify the referred object in the current frame by leveraging multi-frame observations. We propose TrackTeller, a temporal multimodal grounding framework that integrates LiDAR-image fusion, language-conditioned decoding, and temporal reasoning in a unified architecture. TrackTeller constructs a shared UniScene representation aligned with textual semantics, generates language-aware 3D proposals, and refines grounding decisions using motion history and short-term dynamics. Experiments on the NuPrompt benchmark demonstrate that TrackTeller consistently improves language-grounded tracking performance, outperforming strong baselines with a 70% relative improvement in Average Multi-Object Tracking Accuracy and a 3.15-3.4 times reduction in False Alarm Frequency.

Figures (8)

• Figure 1: Key challenges of temporal 3D language grounding. (I) Complex multi-sensor fusion: grounding requires coherent alignment of LiDAR geometry, camera semantics, and language cues. (II) Temporal ambiguity: many expressions reference recent motion or behavior, requiring multi-frame reasoning.
• Figure 2: Overview of TrackTeller. LiDAR and multi-view images are fused into UniScene tokens, which are aligned with language for 3D proposal decoding. A temporal reasoning module enriches proposals with motion history and produces the final grounding score.
• Figure 3: Number of referred targets per prompt.
• Figure 4: Percentage of referred targets per category.
• Figure 5: Performance comparison with representative baselines across filtering thresholds. As the filtering threshold increases, the number of effective candidate targets decreases sharply, which in turn leads to a notable degradation in tracking performance. In practice, threshold values larger than 0.3 substantially reduce overall accuracy. Therefore, we evaluate all methods under threshold values ranging from 0 to 0.3.