Enhancing Self-Supervised Fine-Grained Video Object Tracking with Dynamic Memory Prediction

TL;DR

The paper tackles the challenge of self-supervised fine-grained video object tracking under occlusion and fast motion by enabling multiple reference frames to participate directly in frame reconstruction. It introduces Dynamic Memory Prediction (DMP), consisting of a Reference Frame Memory Engine that maintains Short-Term and Long-Term memories and a Bidirectional Target Prediction Network that propagates target information across frames via cluster-based region alignment. The approach achieves state-of-the-art performance among self-supervised methods on DAVIS-17 video segmentation and JHMDB pose keypoint tracking, approaching supervised methods in several metrics (e.g., mean J&F of $76.4$ on DAVIS-17). Its decoder-free design and plug-and-play nature suggest broad applicability to various video-processing tasks without semantic annotations, while effectively mitigating occlusion and rapid motion challenges.

Abstract

Successful video analysis relies on accurate recognition of pixels across frames, and frame reconstruction methods based on video correspondence learning are popular due to their efficiency. Existing frame reconstruction methods, while efficient, neglect the value of direct involvement of multiple reference frames for reconstruction and decision-making aspects, especially in complex situations such as occlusion or fast movement. In this paper, we introduce a Dynamic Memory Prediction (DMP) framework that innovatively utilizes multiple reference frames to concisely and directly enhance frame reconstruction. Its core component is a Reference Frame Memory Engine that dynamically selects frames based on object pixel features to improve tracking accuracy. In addition, a Bidirectional Target Prediction Network is built to utilize multiple reference frames to improve the robustness of the model. Through experiments, our algorithm outperforms the state-of-the-art self-supervised techniques on two fine-grained video object tracking tasks: object segmentation and keypoint tracking.

Figures (6)

• Figure 1: Performance comparison over DAVIS$_{17}$perazzi2016benchmarkval. The self-supervised method is positioned on the far left of the figure, while the supervised method is located on the right side. Ours surpasses all existing self-supervised methods ($\mathcal{J} \& \mathcal{F}(Mean)$ : 76.4), and is on par with many fully-supervised ones trained with massive annotations.
• Figure 2: The model architecture of DMP. In our encoder-only model, reference frames are dynamically placed in Long-Term or Short-Term Memory. Frames in different memory banks will perform forward or backward target prediction with the target frame separately to obtain the best tracking results.
• Figure 3: Our Frame Region Clustering Model. Blocks of the same color indicate that these regions belong to the same cluster.
• Figure 4: Backward Target Prediction.The candidate regions of pixels in the target frame are restricted to the same color blocks in the reference frames.
• Figure 5: The visualization results of different tasks on DAVIS$_{17}$perazzi2016benchmarkval (a) and JHMDBjhuang2013towardsval (b).