Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

S.T.A.R.-Track: Latent Motion Models for End-to-End 3D Object Tracking with Adaptive Spatio-Temporal Appearance Representations

Simon Doll, Niklas Hanselmann, Lukas Schneider, Richard Schulz, Markus Enzweiler, Hendrik P. A. Lensch

TL;DR

STAR-TRACK is proposed, which uses a novel latent motion model (LMM) to additionally adjust object queries to account for changes in viewing direction and lighting conditions directly in the latent space, while still modeling the geometric motion explicitly.

Abstract

Following the tracking-by-attention paradigm, this paper introduces an object-centric, transformer-based framework for tracking in 3D. Traditional model-based tracking approaches incorporate the geometric effect of object- and ego motion between frames with a geometric motion model. Inspired by this, we propose S.T.A.R.-Track, which uses a novel latent motion model (LMM) to additionally adjust object queries to account for changes in viewing direction and lighting conditions directly in the latent space, while still modeling the geometric motion explicitly. Combined with a novel learnable track embedding that aids in modeling the existence probability of tracks, this results in a generic tracking framework that can be integrated with any query-based detector. Extensive experiments on the nuScenes benchmark demonstrate the benefits of our approach, showing state-of-the-art performance for DETR3D-based trackers while drastically reducing the number of identity switches of tracks at the same time.

S.T.A.R.-Track: Latent Motion Models for End-to-End 3D Object Tracking with Adaptive Spatio-Temporal Appearance Representations

TL;DR

STAR-TRACK is proposed, which uses a novel latent motion model (LMM) to additionally adjust object queries to account for changes in viewing direction and lighting conditions directly in the latent space, while still modeling the geometric motion explicitly.

Abstract

Following the tracking-by-attention paradigm, this paper introduces an object-centric, transformer-based framework for tracking in 3D. Traditional model-based tracking approaches incorporate the geometric effect of object- and ego motion between frames with a geometric motion model. Inspired by this, we propose S.T.A.R.-Track, which uses a novel latent motion model (LMM) to additionally adjust object queries to account for changes in viewing direction and lighting conditions directly in the latent space, while still modeling the geometric motion explicitly. Combined with a novel learnable track embedding that aids in modeling the existence probability of tracks, this results in a generic tracking framework that can be integrated with any query-based detector. Extensive experiments on the nuScenes benchmark demonstrate the benefits of our approach, showing state-of-the-art performance for DETR3D-based trackers while drastically reducing the number of identity switches of tracks at the same time.
Paper Structure (12 sections, 5 equations, 4 figures, 6 tables)

This paper contains 12 sections, 5 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Overall Architecture
  5. Revisiting Multi-Object Tracking
  6. Latent Motion Models
  7. Track Embeddings for implicit Existence Probability
  8. Experiments
  9. Experimental Setup
  10. Comparison to Existing Works
  11. Ablation and Analysis
  12. Conclusion

Figures (4)

  • Figure 1: Visualization of a tracked object for two consecutive frames. Due to ego and object motion the 3D pose and the appearance of the object in the camera images change in scale, viewing angle and lighting condition. We utilize an explicit geometric and a novel latent motion model to compensate for these effects during the prediction step of the tracking pipeline.
  • Figure 2: Star-Track architecture. A joint set of time-independent object queries and track queries of the previous frames is used in a stack of decoder layers that utilize self- and cross-attention blocks to detect and re-identify objects in consecutive time steps. This requires predicting the state of each object in the following frame. Combined with any geometric motion model (blue) the newly proposed latent motion model (green) solves this issue by modeling the spatio-temporal change of a track query in the latent and the 3D geometric space jointly, based on the estimated dynamics.
  • Figure 3: Latent motion model architecture. A geometric transformation consisting of a translation and rotation is applied to the high-dimensional object query by using a sparse latent transformation matrix $K$. We estimate the elements of $K$ with a hyper-network (TfNet) and apply the transformation as an input dependent multiplication, mimicking the behavior of a homogeneous matrix in 3D. Note the sparse block-diagonal shape of the generated matrix.
  • Figure 4: Qualitative results for two consecutive frames on the nuScenes caesar2020nuscenes validation set. Upper row shows predictions and ground truth in top view. Different colors of the predicted objects indicate different object ids. The bottom row shows the predictions projected to the multi-view camera images.