SAM-PM: Enhancing Video Camouflaged Object Detection using Spatio-Temporal Attention

TL;DR

This work addresses video camouflaged object detection (VCOD) by augmenting the Segment Anything Model (SAM) with a lightweight Propagation Module (SAM-PM) that enforces temporal coherence via spatio-temporal cross-attention. The PM comprises a Temporal Fusion Mask Module (TFMM) and a Memory Prior Affinity Module (MPAM); SAM remains frozen, and only the PM is trained, with memory of frames enabling mask propagation across time using under $10^6$ trainable parameters. The approach achieves state-of-the-art results on MoCA-Mask and CAD, showing substantial improvements across standard VCOD metrics while maintaining parameter efficiency. This demonstrates a practical pathway to infusing temporal and domain-specific knowledge into large foundation models for video segmentation, accompanied by open-source code for reproducibility.

Abstract

In the domain of large foundation models, the Segment Anything Model (SAM) has gained notable recognition for its exceptional performance in image segmentation. However, tackling the video camouflage object detection (VCOD) task presents a unique challenge. Camouflaged objects typically blend into the background, making them difficult to distinguish in still images. Additionally, ensuring temporal consistency in this context is a challenging problem. As a result, SAM encounters limitations and falls short when applied to the VCOD task. To overcome these challenges, we propose a new method called the SAM Propagation Module (SAM-PM). Our propagation module enforces temporal consistency within SAM by employing spatio-temporal cross-attention mechanisms. Moreover, we exclusively train the propagation module while keeping the SAM network weights frozen, allowing us to integrate task-specific insights with the vast knowledge accumulated by the large model. Our method effectively incorporates temporal consistency and domain-specific expertise into the segmentation network with an addition of less than 1% of SAM's parameters. Extensive experimentation reveals a substantial performance improvement in the VCOD benchmark when compared to the most recent state-of-the-art techniques. Code and pre-trained weights are open-sourced at https://github.com/SpiderNitt/SAM-PM

Figures (5)

• Figure 1: Comparison of mask predictions between ground truth (GT), SAM-PM (Ours), and SLT-Net.
• Figure 2: Overall framework of SAM-PM consisting of SAM(frozen), PM(trainable) and Memory. Instead of adding the image and mask directly, we incorporate their embeddings into Memory. This prevents redundant encoding when predicting future masks. One significant advantage of this architecture is that we only need to encode the input frame once, regardless of the number of objects we aim to track.
• Figure 3: Overview of our Propagation Module consisting of TFMM and MPAM. It's important to observe that we utilize separate positional encoding for each module to provide greater flexibility within the model. Both positional encodings come with trainable parameters, enabling the model to regulate the extent of positional encoding applied to them.
• Figure 4: Visual comparison of some recent VCOD models with ours on MoCA-Mask
• Figure 5: Visual comparison of some recent VCOD models with ours on CAD Dataset.