Dynamic in Static: Hybrid Visual Correspondence for Self-Supervised Video Object Segmentation

TL;DR

This work tackles the data-hungry nature of video object segmentation by proposing HVC, a self-supervised framework that learns hybrid static-dynamic visual correspondence purely from static images. By cropping overlapped views and training with an asymmetric online/target encoder plus a lightweight pseudo-dynamic generator, HVC enforces both static and pseudo-dynamic consistency through a joint loss, while avoiding cross-frame reconstruction. Ablation and extensive benchmarking show that static and dynamic cues jointly improve dense correspondence, yielding state-of-the-art results among self-supervised VOS methods on DAVIS and YouTube-VOS, with strong performance on downstream label propagation tasks and impressive efficiency (≈2 hours training, ≈16 GB memory). The approach offers a scalable alternative to video-based self-supervised learning, reducing data requirements and computation while maintaining competitive or superior segmentation capabilities across challenging datasets.

Abstract

Conventional video object segmentation (VOS) methods usually necessitate a substantial volume of pixel-level annotated video data for fully supervised learning. In this paper, we present HVC, a \textbf{h}ybrid static-dynamic \textbf{v}isual \textbf{c}orrespondence framework for self-supervised VOS. HVC extracts pseudo-dynamic signals from static images, enabling an efficient and scalable VOS model. Our approach utilizes a minimalist fully-convolutional architecture to capture static-dynamic visual correspondence in image-cropped views. To achieve this objective, we present a unified self-supervised approach to learn visual representations of static-dynamic feature similarity. Firstly, we establish static correspondence by utilizing a priori coordinate information between cropped views to guide the formation of consistent static feature representations. Subsequently, we devise a concise convolutional layer to capture the forward / backward pseudo-dynamic signals between two views, serving as cues for dynamic representations. Finally, we propose a hybrid visual correspondence loss to learn joint static and dynamic consistency representations. Our approach, without bells and whistles, necessitates only one training session using static image data, significantly reducing memory consumption ($\sim$16GB) and training time ($\sim$\textbf{2h}). Moreover, HVC achieves state-of-the-art performance in several self-supervised VOS benchmarks and additional video label propagation tasks.

Figures (12)

• Figure 1: Self-supervised VOS performance comparison on DAVIS$_{17}$pont2017val-set. We adopt the same hardware device and training data xu2018youtube for a fair comparison. Bubble size indicates the size of training images.Taking less training time, our model outperforms state-of-the-art methods across all scales of data volume. HVC trained with 95K images achieves comparable performance to LIIR li2022liir delivered with 470K ones.
• Figure 2: Learned representation visualization from HVC without any supervision. Our proposed self-supervised hybrid visual correspondence learning highlights salient regions, suggesting the suitability of the HVC-learned representations for dense tasks such as video object segmentation.
• Figure 3: Architecture of the proposed HVC. Given an image $\bm{I}$, its pair of views $\bm{I}_{1}$ and $\bm{I}_{2}$ are obtained by crop-resize transformation. Two views are fed to the online and target networks, $\mathcal{F}_{\bm{\theta}_{\rm{online}}}$ and $\mathcal{F}_{\bm{\theta}_{\rm{target}}}$ (§\ref{['sec:3.1']}). The online network contains the additional projector and predictor heads to acquire feature maps $\bm{F}_{1}$ and the target network appends only projector heads to receive feature maps $\bm{F}_{2}$. The pseudo-dynamic signal generation module receives $\bm{F}_{1}$ and $\bm{F}_{2}$ and outputs forward and backward pseudo-dynamic signals, $\bm{M}_{1}$ and $\bm{M}_{2}$. Inter-feature and inter-dynamic similarities are combined with a positive sample mask to yield hybrid similarity (taking the negative, i.e., final loss, see §\ref{['sec:3.2']} for details).
• Figure 4: (a) Illustration of static and dynamic visual correspondence. (b) Visualization of mask propagation.
• Figure 5: Statistics of encoder weights (§\ref{['sec:3.2']}) at each layer of ResNet-18. The weights learned by dynamic, static and hybrid approaches are displayed from left to right in each violin plot.