Implicit Motion-Compensated Network for Unsupervised Video Object Segmentation

TL;DR

This work tackles unsupervised video object segmentation by integrating appearance and motion cues without explicit optical-flow estimation. The IMCNet architecture combines an affinity-based ACM, an attention-driven APM, and a cascade deformable MCM to align and fuse information across multiple frames at the feature level. A joint training strategy exposing the model to both UVOS and salient-object data enhances local discriminability while preserving temporal consistency. Empirical results on DAVIS_16 and YouTube-Objects show competitive accuracy with a favorable speed-parameter trade-off, confirming the effectiveness of implicit motion compensation and top-down attention in UVOS. The approach offers a scalable, efficient alternative to flow-based multi-frame methods with strong practical applicability in video analysis tasks.

Abstract

Unsupervised video object segmentation (UVOS) aims at automatically separating the primary foreground object(s) from the background in a video sequence. Existing UVOS methods either lack robustness when there are visually similar surroundings (appearance-based) or suffer from deterioration in the quality of their predictions because of dynamic background and inaccurate flow (flow-based). To overcome the limitations, we propose an implicit motion-compensated network (IMCNet) combining complementary cues ($\textit{i.e.}$, appearance and motion) with aligned motion information from the adjacent frames to the current frame at the feature level without estimating optical flows. The proposed IMCNet consists of an affinity computing module (ACM), an attention propagation module (APM), and a motion compensation module (MCM). The light-weight ACM extracts commonality between neighboring input frames based on appearance features. The APM then transmits global correlation in a top-down manner. Through coarse-to-fine iterative inspiring, the APM will refine object regions from multiple resolutions so as to efficiently avoid losing details. Finally, the MCM aligns motion information from temporally adjacent frames to the current frame which achieves implicit motion compensation at the feature level. We perform extensive experiments on $\textit{DAVIS}_{\textit{16}}$ and $\textit{YouTube-Objects}$. Our network achieves favorable performance while running at a faster speed compared to the state-of-the-art methods.

Figures (13)

• Figure 1: Illustration of the failure case of existing UVOS methods. The images with a green border as the input frames fed into the models $F(\theta)$ to produce object masks inside the orange border. The objects in the green, blue and red ellipse box are respectively similar surroundings, misleading objects, and motion artifacts. Note that target objects (ground-truth) are highlighted by green translucent masks.
• Figure 2: The overview of the proposed IMCNet. The $2N+1$ consecutive frames $\{I_{i}\}_{i=t-N\Delta t}^{t+N\Delta t}$ are first fed into the FEM to extract embedding of each frame as $V_{i}^{2} \sim V_{i}^{5}$. Then, the ACM computes the affinity of each feature $V_{i}^{5}$ that summarizes the global dependence among input frames $I_{i}$. The attention enhanced features $Z_{i}$ are further fed into the APM to transmit global dependence via a top-down manner. Finally, the output of the top-down decoder is passed to the MCM to facilitate obtaining the final segmentation result $\hat{M}_{t}$. Here, the three key encoders for the center and neighboring frames are parameter-shared.
• Figure 3: Illustration of affinity computing process. Taking center moment $t$ as an example, the key features $K_{i}$ are flattened into matrices and used to compute the affinity matrix $S$ via Eq. \ref{['eq:ac:2']}. Then, the affinity matrix $S$ is normalized via Eq. \ref{['eq:ac:3']} as attention weight (i.e., $S^{r}$). Finally, the value features $V_{t}^{5}$ are post-multiplied by the $S^{r}$ to compute the attention enhanced feature $Z_{t}$ (Eq. \ref{['eq:ac:4']}).
• Figure 4: The proposed APM. Attention propagation operation is implemented by Eq. \ref{['eq:apm:1']}. The computational graph of center moment $t$ is taken as an example. Here, '$\otimes$', '$\circledast$', '$\oplus$', and '$\mathrm{Up}$' indicate element-wise multiplication, element-wise multiplication with broadcasting, element-wise addition, and bilinear upsampling, respectively. The '$\mathrm{GAP}$' denotes the global average pooling operation.
• Figure 5: In the architecture of cascading alignment, '$\mathrm{C}$' denotes concatenation. The alignment process will iterate 4 times (i.e., $L=4$), the initial aligned feature $f_{\mathrm{aligned}}^{0}$ is $f_{\mathrm{nbr}^{\left\{+, -\right\}}}$. The output of the previous alignment module will be concatenated with $f_{ref}$ and fed into the next alignment module.