IDPro: Flexible Interactive Video Object Segmentation by ID-queried Concurrent Propagation

TL;DR

IDPro tackles interactive video object segmentation by enabling users to annotate multiple frames simultaneously and process multiple objects in parallel. It introduces two novel modules—the Across-Frame Interaction Module (AFI) and the Concurrent Propagation Module—with an across-round memory and an ID-queried decoder to efficiently propagate scribble guidance across frames and rounds. Through AFI’s Across-Frame Attention and FrameEnhancer, and a truncated re-propagation strategy, IDPro achieves a new state-of-the-art on DAVIS 2017 with $89.6\%$ $\mathcal{J\&F}@60$, while the R50 variant delivers over a 3× speedup in challenging multi-object scenarios. An interactive GUI supports multi-frame input and multi-object annotations, enabling practical, real-time use. Overall, IDPro significantly improves user experience by combining flexible input, efficient multi-object propagation, and robust memory mechanisms for iVOS.

Abstract

Interactive Video Object Segmentation (iVOS) is a challenging task that requires real-time human-computer interaction. To improve the user experience, it is important to consider the user's input habits, segmentation quality, running time and memory consumption.However, existing methods compromise user experience with single input mode and slow running speed. Specifically, these methods only allow the user to interact with one single frame, which limits the expression of the user's intent.To overcome these limitations and better align with people's usage habits, we propose a framework that can accept multiple frames simultaneously and explore synergistic interaction across frames (SIAF). Concretely, we designed the Across-Frame Interaction Module that enables users to annotate different objects freely on multiple frames. The AFI module will migrate scribble information among multiple interactive frames and generate multi-frame masks. Additionally, we employ the id-queried mechanism to process multiple objects in batches. Furthermore, for a more efficient propagation and lightweight model, we design a truncated re-propagation strategy to replace the previous multi-round fusion module, which employs an across-round memory that stores important interaction information. Our SwinB-SIAF achieves new state-of-the-art performance on DAVIS 2017 (89.6%, J&F@60). Moreover, our R50-SIAF is more than 3 faster than the state-of-the-art competitor under challenging multi-object scenarios.

Figures (9)

• Figure 1: IDPro enables users to interact with multiple frames at once, whereas the SOTA method can only accept one single frame. Moreover, IDPro processes multiple objects simultaneously, leading to faster model performance and reduced computing resources.
• Figure 2: The overview of IDPro (§\ref{['sec:overview']}). IDPro can be formulated as interaction-propagation. During interaction round $r$, the Across-Frame Interaction Module enables users to annotate multiple frames and generate the corresponding masks. Next, the Concurrent Propagation Module is designed to generate the masks for the non-interactive frames. To preserve crucial interaction information, we update the across-round memory after each round. Then we generate masks with the updated memory by the ID-queried decoder. We further introduce a re-propagation strategy to address conflicts in multi-round propagation.
• Figure 3: The implementation of Across-Frame Interaction Module (AFI; §\ref{['sec:s2m']}). To capture the multi-scale features within each frame and the temporal information across frames, we design an across-frame attention (AFA). To enhance the modeling of the current frame, we develop the FrameEnhancer Network. The premask at the top right of the image denotes the previous mask.
• Figure 4: The attention map of Across-Frame Attention (AFA; §\ref{['sec:afa']}). Continuing Fig. \ref{['fig:s2m']}, showing the S2M in MiVOS that accepts only one single frame is insufficient. Suppose we aim to predict the mask of $t_{j}$-th frame. (d) The mask produced by S2M only refers to the $t_{j}$-th frame loss of the handlebar. To address this challenge, we utilize across-frame features to generate an attention map (b), which represents the handlebar's attention position in the $t_{j}$-th frame (e) on the $t_{i}$-th frame (c). Thus, the mask (e) produced by IDPro contains the information of the other interacted frame (e.g. the handlebar information at $t_{i}$-th frame).
• Figure 5: The illustration of Truncated Propagation and Re-propagation Strategy (§\ref{['sec:prop']}). To overcome the conflicts, we propose the truncated propagation that propagates the masks of the interacted frames to the midpoint of the two frames. For re-propagation, all frames without interaction will be re-propagated with the more robust across-round memory that we designed.