Self-supervised Video Object Segmentation with Distillation Learning of Deformable Attention

TL;DR

The paper tackles video object segmentation under temporal changes by introducing deformable attention to adapt attention maps across frames and a self-supervised distillation framework to train a lightweight student from a large teacher. It combines a gated deformable propagation module with a knowledge distillation objective that transfers both intermediate attention maps and logits, using Centered Kernel Alignment for attention alignment and intra-/inter-object relations for logits. Empirical results on DAVIS 2016/2017 and YouTube-VOS 2018/2019 demonstrate state-of-the-art performance with significantly improved memory efficiency and real-time inference on low-power devices. The approach enhances robustness to motion and deformation while maintaining competitive accuracy, showing strong potential for practical deployments in resource-constrained settings.

Abstract

Video object segmentation is a fundamental research problem in computer vision. Recent techniques have often applied attention mechanism to object representation learning from video sequences. However, due to temporal changes in the video data, attention maps may not well align with the objects of interest across video frames, causing accumulated errors in long-term video processing. In addition, existing techniques have utilised complex architectures, requiring highly computational complexity and hence limiting the ability to integrate video object segmentation into low-powered devices. To address these issues, we propose a new method for self-supervised video object segmentation based on distillation learning of deformable attention. Specifically, we devise a lightweight architecture for video object segmentation that is effectively adapted to temporal changes. This is enabled by deformable attention mechanism, where the keys and values capturing the memory of a video sequence in the attention module have flexible locations updated across frames. The learnt object representations are thus adaptive to both the spatial and temporal dimensions. We train the proposed architecture in a self-supervised fashion through a new knowledge distillation paradigm where deformable attention maps are integrated into the distillation loss. We qualitatively and quantitatively evaluate our method and compare it with existing methods on benchmark datasets including DAVIS 2016/2017 and YouTube-VOS 2018/2019. Experimental results verify the superiority of our method via its achieved state-of-the-art performance and optimal memory usage.

Figures (14)

• Figure 1: (a) From left to right: input frame and ground-truth segmentation masks, distilled feature map and segmentation masks by the distillation strategy used in MobileVOS miles2023mobilevos, distilled feature map and segmentation masks by our method. (b) Segmentation accuracy ($\mathcal{J}$&$\mathcal{F}$) of MobileVOS miles2023mobilevos and our method with different numbers of query objects in DAVIS-17 val (see more details in Section \ref{['sec:experiments']}).
• Figure 2: Summary of our proposed VOS method. (a) Overview of our knowledge distillation method. The teacher model transfers intermediate attention maps to the student model. This transfer is enforced by a CKA-based loss $\mathcal{L}_{att}$. At the same time, probability distributions of logits are transferred using intra-object and inter-object losses $\mathcal{L}_{intra}$ and $\mathcal{L}_{inter}$. Both the teacher and student models make use of Gated Propagation Module (GPM) yang2022decoupling, aiming to propagate spatio-temporal information across frames via the attention mechanism. (b) Our proposed Gated Deformable Attention function, which is used to implement the GPM. (c) Deformable attention module, which is used to replace the vanilla attention in the Gated deformable attention function.
• Figure 3: Qualitative results of our method and a baseline (a DeAOTT model with the vanilla attention trained using the standard KD, i.e., only logit layers are transferred). As shown, compared with the baseline, our method can maintain the association of the objects and their IDs across frames (see frame 36 in the 1st and 2nd row). Our method also tends to be aware of parts of the same object (see frame 50 in the 3rd and 4th row). We hypothesize this success is due to the cross-frame adaptivity of deformable attention over its counterpart. More results are provided in our supplementary material.
• Figure 4: Comparison of self-supervised VOS methods in terms of segmentation accuracy ($\mathcal{J}\&\mathcal{F}$) and memory footprint on DAVIS-17 Val dataset.
• Figure 5: Convergence analysis of the loss functions.