Learning segmentation from point trajectories

TL;DR

We address unsupervised video object segmentation by exploiting long-term point trajectories as supervisory signals in addition to optical flow. The core idea is a subspace-clustering–inspired loss that encourages trajectories belonging to the same object to lie in a low-rank subspace, implemented as a trajectory loss on per-object trajectory matrices, optionally paired with a flow-based loss. The method jointly trains a segmentation network with flow and trajectory losses, including a temporal smoothing term, and uses a rank parameter $r$ to control the low-rank constraint. Feasibility studies on synthetic data validate the loss behavior, and experiments on MOVi-F and real VOS benchmarks show state-of-the-art performance without annotations, demonstrating that long-term motion information complements dense flow for robust segmentation. Overall, the work demonstrates that long-term motion supervision can scale to dataset-wide training and yield strong unsupervised segmentation performance with practical impact for motion-aware video analysis.

Abstract

We consider the problem of segmenting objects in videos based on their motion and no other forms of supervision. Prior work has often approached this problem by using the principle of common fate, namely the fact that the motion of points that belong to the same object is strongly correlated. However, most authors have only considered instantaneous motion from optical flow. In this work, we present a way to train a segmentation network using long-term point trajectories as a supervisory signal to complement optical flow. The key difficulty is that long-term motion, unlike instantaneous motion, is difficult to model -- any parametric approximation is unlikely to capture complex motion patterns over long periods of time. We instead draw inspiration from subspace clustering approaches, proposing a loss function that seeks to group the trajectories into low-rank matrices where the motion of object points can be approximately explained as a linear combination of other point tracks. Our method outperforms the prior art on motion-based segmentation, which shows the utility of long-term motion and the effectiveness of our formulation.

Figures (23)

• Figure 1: Illustrative 2D example for the low-rank nature of $P_k$. A triangle undergoes rigid rotation over three frames. As the rate of rotation is not constant, the flow vectors and point positions are difficult to model. However, the point $p$ is part of the triangle and can be expressed as a combination of the three vertices at an appropriate time. Thus, the last column of $P_k$ is linearly dependent, and $P_k$ is rank deficient. Any points in the triangle could be included in $P_k$ without increasing its rank.
• Figure 2: Overview of our approach. We self-supervise a segmentation network, i.e., without access to mask annotations, using both short-term motion information (optical flow) and long-term motion (point trajectories). We design a loss function that encourages the segmentation network to cluster regions where trajectories form low-rank-$r$ groups, which should align well with objects. Off-the-shelf methods are used to estimate optical flow and point trajectories given a dataset of videos.
• Figure 3: Feasibility analysis of $\mathcal{L}_t$. Using a synthetic sequence (left), we vary the amount of noise $\eta$ injected into the mask, the temperature $\tau$ of the mask logits and plot the loss value as a function of the mask under/over segmentation. The plots show that the loss is reduced in low-noise, low-entropy settings and penalises both over- and under-segmentation.
• Figure 4: Qualitative comparison of our results on DAVIS with RCF which uses higher resolution and multi-stage training. Our method contains slightly better boundaries, does not segment shadows and separates water ripples from the swan.
• Figure 5: Qualitative comparison of our results on SegTrackv2 with RCF which uses higher resolution and multi-stage training. Our method contains slightly better boundaries and segments more whole objects.