Learning Appearance and Motion Cues for Panoptic Tracking

TL;DR

This work addresses panoptic tracking in dynamic scenes by integrating appearance- and motion-guided cues within an end-to-end MAPT architecture. MAPT deploys a shared backbone with four interconnected heads—semantic segmentation, instance tracking, motion tracking, and appearance tracking—plus a fusion module to produce temporally coherent panoptic outputs. The Motion head enables semantic-feature-driven mask propagation, while the Appearance head learns motion-enhanced embeddings, with a two-stage fusion that robustly associates objects across frames. Evaluations on KITTI-STEP and MOTChallenge-STEP demonstrate state-of-the-art Panoptic And Tracking (PAT) performance and strong segmentation metrics, with ablations confirming the complementary benefits of jointly learning appearance and motion cues and providing public code for reproducibility.

Abstract

Panoptic tracking enables pixel-level scene interpretation of videos by integrating instance tracking in panoptic segmentation. This provides robots with a spatio-temporal understanding of the environment, an essential attribute for their operation in dynamic environments. In this paper, we propose a novel approach for panoptic tracking that simultaneously captures general semantic information and instance-specific appearance and motion features. Unlike existing methods that overlook dynamic scene attributes, our approach leverages both appearance and motion cues through dedicated network heads. These interconnected heads employ multi-scale deformable convolutions that reason about scene motion offsets with semantic context and motion-enhanced appearance features to learn tracking embeddings. Furthermore, we introduce a novel two-step fusion module that integrates the outputs from both heads by first matching instances from the current time step with propagated instances from previous time steps and subsequently refines associations using motion-enhanced appearance embeddings, improving robustness in challenging scenarios. Extensive evaluations of our proposed \netname model on two benchmark datasets demonstrate that it achieves state-of-the-art performance in panoptic tracking accuracy, surpassing prior methods in maintaining object identities over time. To facilitate future research, we make the code available at http://panoptictracking.cs.uni-freiburg.de

Figures (5)

• Figure 1: Proposed MAPT framework integrating appearance and motion cues for panoptic tracking. Our approach improves tracking robustness by leveraging both appearance and motion features, enabling better spatio-temporal scene understanding in dynamic environments.
• Figure 2: Our MAPT architecture is composed of a shared backbone and four interconnected heads for semantic segmentation, instance segmentation, and motion-based and appearance-based object tracking. Our motion head employs multi-scale deformable convolutions to capture rich semantic and motion features, while the appearance head focuses on learning instance-specific visual representations. These two heads complement each other, as motion cues help in scenarios where appearance alone is ambiguous, while appearance features provide stability when motion is unreliable. By combining the outputs with our fusion block, enhance the robustness of panoptic tracking in dynamic environments.
• Figure 3: Qualitative comparisons of panoptic tracking from our proposed MAPT (first and second rows) with Video-Knet (last row). Each column shows a different frame in a video. These results demonstrate that our MAPT method effectively maintains object identity and shape consistency through occlusions.
• Figure 4: We compared our proposed MAPT model with Video-Knet by analyzing three video frames, highlighting how occlusion-related visual distortion affects prediction accuracy. These results highlight that our MAPT model better preserves track identity and object shape under occlusions, leading to more consistent panoptic tracking
• Figure 5: Comparison between our proposed MAPT and Video-Knet based on close-up analysis of three frames in a video presenting visual distortion due to occlusion that affects the predictions. These observations highlight the advantage of our MAPT method in handling occlusions and preserving object integrity.