SLAck: Semantic, Location, and Appearance Aware Open-Vocabulary Tracking

TL;DR

SLAck tackles open-vocabulary MOT by jointly fusing semantic, location, and appearance cues at the earliest association stage. It introduces a Spatial-Temporal Object Graph (STOG) that enables intra-frame and inter-frame reasoning, with a differentiable Sinkhorn-based objective and Detection Aware Training to handle incomplete TAO annotations. The method achieves state-of-the-art performance on open-vocabulary benchmarks (OVTrack) and TAO TETA, particularly in association accuracy for novel classes, demonstrating strong generalization. The framework is end-to-end, detector-free of post-hoc fusion heuristics, and comes with open-source code, offering a practical approach for robust open-world tracking.

Abstract

Open-vocabulary Multiple Object Tracking (MOT) aims to generalize trackers to novel categories not in the training set. Currently, the best-performing methods are mainly based on pure appearance matching. Due to the complexity of motion patterns in the large-vocabulary scenarios and unstable classification of the novel objects, the motion and semantics cues are either ignored or applied based on heuristics in the final matching steps by existing methods. In this paper, we present a unified framework SLAck that jointly considers semantics, location, and appearance priors in the early steps of association and learns how to integrate all valuable information through a lightweight spatial and temporal object graph. Our method eliminates complex post-processing heuristics for fusing different cues and boosts the association performance significantly for large-scale open-vocabulary tracking. Without bells and whistles, we outperform previous state-of-the-art methods for novel classes tracking on the open-vocabulary MOT and TAO TETA benchmarks. Our code is available at \href{https://github.com/siyuanliii/SLAck}{github.com/siyuanliii/SLAck}.

Figures (8)

• Figure 1: Unlike conventionally pedestrian tracking such as MOT20 MOT20, open-world objects show high dynamicity on shape and motion, which poses significant challenges for motion-based trackers.
• Figure 2: (a) Current MOT methods mainly use motion or appearance cues for tracking. Motion-based trackers typically assume linear motion, employing Kalman Filters, which do not account for the complex, non-linear movements observed in various classes, leading to tracking failures. Appearance-based tracking, which overlooks location information, may confuse targets with similar looks. Semantic cues are either ignored or used to group instances within the same class as a gating function, which usually leads to errors due to poor classification accuracy in the open world. Hybrid trackers combine all three cues—semantic, location, and appearance in a heuristic way which suffers the high dynamicity of the open world. (b) Our method jointly fuses the semantic, location, and appearance cues in the early matching stage and yields a single joint similarity matrix that can be directly optimized and trained end-to-end for association.
• Figure 3: Comparison of different strategies on utilizing semantic cues in open-vocabulary MOT. Instead of leveraging hard grouping or soft grouping TETer, our method integrates semantic cues at the early stage to improve the association accuracy.
• Figure 4: Overall pipeline of SLAck. SLAck first extracts semantic, location, and appearance cues from pre-trained detectors, and then constructs the Spatial-Temporal Object Graph to fuse these cues and model the object dynamics for tracking.
• Figure 5: Unlike conventionally pedestrian tracking such as MOT20 MOT20, open-world objects show high dynamicity on shape and motion, which poses significant challenges for motion-based trackers. We demonstrate this quantitatively on Fig. \ref{['fig:kde_taovsmot']}.