Memory-Efficient Continual Learning Object Segmentation for Long Video

TL;DR

This paper tackles memory limitations in semi-supervised online video object segmentation (VOS) for long videos, where storing all past frames is impractical. It recasts Online VOS as online continual learning and introduces two memory-efficient modules: GRCL, which uses a gated-regularizer memory to preserve previously learned object representations, and RMSCL, which selects a compact, diverse subset of memory via reconstruction-based criteria. A Hybrid approach combines GRCL and RMSCL to exploit both guarantees of stability and selective rehearsal. Across long-video benchmarks, the methods yield averaging gains above 8% in J&F and enhance robustness while preserving short-video performance on DAVIS16, DAVIS17, and YouTube-VOS18, all without retraining the offline encoder/decoder components.

Abstract

Recent state-of-the-art semi-supervised Video Object Segmentation (VOS) methods have shown significant improvements in target object segmentation accuracy when information from preceding frames is used in segmenting the current frame. In particular, such memory-based approaches can help a model to more effectively handle appearance changes (representation drift) or occlusions. Ideally, for maximum performance, Online VOS methods would need all or most of the preceding frames (or their extracted information) to be stored in memory and be used for online learning in later frames. Such a solution is not feasible for long videos, as the required memory size grows without bound, and such methods can fail when memory is limited and a target object experiences repeated representation drifts throughout a video. We propose two novel techniques to reduce the memory requirement of Online VOS methods while improving modeling accuracy and generalization on long videos. Motivated by the success of continual learning techniques in preserving previously-learned knowledge, here we propose Gated-Regularizer Continual Learning (GRCL), which improves the performance of any Online VOS subject to limited memory, and a Reconstruction-based Memory Selection Continual Learning (RMSCL), which empowers Online VOS methods to efficiently benefit from stored information in memory. We also analyze the performance of a hybrid combination of the two proposed methods. Experimental results show that the proposed methods are able to improve the performance of Online VOS models by more than 8%, with improved robustness on long-video datasets while maintaining comparable performance on short-video datasets such as DAVIS16, DAVIS17, and YouTube-VOS18.

Figures (12)

• Figure 1: The proposed Online VOS framework, with adopted Gated-Regularized Continual Learning (GRCL): At time $t$, the overall gated-regularizer map $\mathbf{G}^{t-1}$ is calculated using the stored gated maps in the gated-regularizer memory $\mathcal{M}^{t-1}_G$ and regularizes the process of updating $\mathrm{C}^t$. Finally, $\mathcal{M}^{t-1}_G$ is updated and forms $\mathcal{M}^{t}_G$ using the calculated $G^{t}$.
• Figure 2: The proposed Online VOS framework with augmented Reconstruction-based Memory Selection Continual Learning (RMSCL). At the current time $t$, three samples associated to three positive $\psi$ are selected using a reconstruction based (Lasso) optimization.
• Figure 3: GPU memory usage of XMem, LWL and JOINT when processing $2416$ frames of the $\boldsymbol{blueboy}$ video in the long video dataset liang2020video. As shown, the GPU memory usage of XMem increases significantly over time, whereas LWL and JOINT have a fixed GPU memory usage.
• Figure 4: Performance comparison of competing methods as a function of memory and target model update step sizes, ($\Delta_{\mathrm{C}}=\Delta_{\mathcal{M}}$), on the Long Videos dataset liang2020video. The left figure shows the average $\mathcal{J}\& \mathcal{F}$ of applying different proposed methods on LWL and the right one shows the performance of the same methods on JOINT. The green line shows the performance of LWL and JOINT without updating their target model on the memory.
• Figure 5: Qualitative comparison of the competing frameworks in the context of the long-video dataset. The associated frame number for each image is shown along the bottom. The leftmost column shows the given mask $Y_g$, which is the same for all methods. The results show that the proposed GRCL, when augmenting the baseline frameworks (LWL and JOINT), can lead to better performance against representation drift. Additionally, the frameworks based on RMSCL (LWL-RMSCL, JOINT-RMCSL) are less vulnerable to the distribution changes which take place in long video sequences. Finally, as shown in the figure, LWL-Hybrid has the best performance among all the proposed methods.