Offline Tracking with Object Permanence

TL;DR

This work tackles offline auto-labeling under long occlusions by introducing an object-permanence inspired offline tracking pipeline. It combines an online tracker with a Re-ID module that uses motion dynamics and a vectorized lane-map prior, and a track completion module that interpolates occluded segments with variable horizon queries. The approach demonstrates state-of-the-art performance on nuScenes MOT benchmarks and shows strong gains in both real and pseudo-occlusion scenarios, illustrating its potential as a plugin for improved offline labeling. By leveraging lane-map priors and flexible occlusion horizons, the method robustly recovers occluded trajectories, enabling more complete and accurate 3D MOT data for downstream labeling tasks.

Abstract

To reduce the expensive labor cost for manual labeling autonomous driving datasets, an alternative is to automatically label the datasets using an offline perception system. However, objects might be temporally occluded. Such occlusion scenarios in the datasets are common yet underexplored in offline auto labeling. In this work, we propose an offline tracking model that focuses on occluded object tracks. It leverages the concept of object permanence which means objects continue to exist even if they are not observed anymore. The model contains three parts: a standard online tracker, a re-identification (Re-ID) module that associates tracklets before and after occlusion, and a track completion module that completes the fragmented tracks. The Re-ID module and the track completion module use the vectorized map as one of the inputs to refine the tracking results with occlusion. The model can effectively recover the occluded object trajectories. It achieves state-of-the-art performance in 3D multi-object tracking by significantly improving the original online tracking result, showing its potential to be applied in offline auto labeling as a useful plugin to improve tracking by recovering occlusions.

Figures (9)

• Figure 1: A brief overview of the offline tracking model. (a) Online tracking result: Each tracklet is represented by a different color (the history tracklet is red). (b) Offline Re-ID: The matched pair of tracklets are red. The unmatched ones are black. (c) Recovered trajectory.
• Figure 2: A brief overview of the map branch. The branch starts with three parallel encoders which encode the future tracklet, lanes and history tracklet respectively. The model then propagates information between tracklets and the lane map by performing attention. Finally, map-based affinity scores are decoded from the tracklet features.
• Figure 3: The network structure of motion affinity branch. The history tracklet encoder is orange, whereas the future tracklets encoder is blue. Three possible future candidates correspond to the three outputted motion affinity scores.
• Figure 4: A brief overview of the track completion model. The model stacks several sub-modules together to sequentially aggregate features and refine the results.
• Figure 5: Qualitative results of the offline tracking model. In each sample, GT boxes are plotted as rectangles. Each GT track is represented by a unique color. Model outputs are depicted with arrows, rather than rectangles, to differentiate from GT tracks. Red and blue arrows indicate history and future tracklets matched by the offline Re-ID model, respectively, while purple arrows show recovered trajectories. Orange dotted lines represent lanes. In the background, the white area is drivable. The gray area is the pedestrian crossing. The red cross is the average position of the ego vehicle during the occlusion.