A2VIS: Amodal-Aware Approach to Video Instance Segmentation

TL;DR

Occlusion poses a major challenge for video instance segmentation and MOT. A2VIS introduces amodal-aware representations and a Spatiotemporal-prior Amodal Mask Head (SAMH) to predict both visible and amodal masks and maintains robust object identities through global instance prototypes. The method jointly detects, segments, and tracks objects end-to-end, achieving state-of-the-art results on VIS/MOT benchmarks and demonstrating stronger occlusion handling than prior amodal or visible-only approaches. This framework offers a practical path toward reliable video understanding in occluded, cluttered scenes.

Abstract

Handling occlusion remains a significant challenge for video instance-level tasks like Multiple Object Tracking (MOT) and Video Instance Segmentation (VIS). In this paper, we propose a novel framework, Amodal-Aware Video Instance Segmentation (A2VIS), which incorporates amodal representations to achieve a reliable and comprehensive understanding of both visible and occluded parts of objects in a video. The key intuition is that awareness of amodal segmentation through spatiotemporal dimension enables a stable stream of object information. In scenarios where objects are partially or completely hidden from view, amodal segmentation offers more consistency and less dramatic changes along the temporal axis compared to visible segmentation. Hence, both amodal and visible information from all clips can be integrated into one global instance prototype. To effectively address the challenge of video amodal segmentation, we introduce the spatiotemporal-prior Amodal Mask Head, which leverages visible information intra clips while extracting amodal characteristics inter clips. Through extensive experiments and ablation studies, we show that A2VIS excels in both MOT and VIS tasks in identifying and tracking object instances with a keen understanding of their full shape.

Figures (9)

• Figure 1: Comparison between existing VIS and the proposed A2VIS. By integrating amodal knowledge, A2VIS perceives the complete trajectory and shape of a target. This contrasts with other VIS methods that do not predict occluded parts, making them inherently susceptible to losing track of the target.
• Figure 2: Overall architecture of the proposed A2VIS. "IP" denotes instance prototypes in this figure. In each clip $\mathcal{V}^k$, the IP Modelling generates the clip-based IP $\mathbf{p}^k$, which is subsequently updated with the global IP $\mathbf{p}^{\mathcal{G}}$ through the IP Update module. The updated $\mathbf{p}^{\mathcal{G}}$ is then used to produce both visible segmentation $\mathbf{M}^k$ and amodal segmentation $\mathbf{A}^k$.
• Figure 3: Network design of Spatiotemporal-prior Amodal Mask Head (SAMH), which takes the frame feature $\mathbf{F}^k$, visible segmentation $\mathbf{M}^k$ and the global instance prototypes $\mathbf{p}^{\mathcal{G}}$ as inputs to generate amodal segmentations $\mathbf{A}^k$ and updates the global instance prototypes $\mathbf{p}^{\mathcal{G}}$. In this figure, "IP" denotes instance prototypes.
• Figure 4: Qualitative results of A2VIS on FISHBOWL dataset (first two rows) and SAILVOS dataset (last two rows).
• Figure 5: Qualitative comparisons of A2VIS with GenVIS-Amodal. Videos are sourced from SAIL-VOS testset.