CamoSAM2: Motion-Appearance Induced Auto-Refining Prompts for Video Camouflaged Object Detection

TL;DR

This work tackles video camouflaged object detection by automatically generating and refining prompts for the Segment Anything Model 2 (SAM2). It introduces Motion-Appearance Prompt Inducer (MAPI) to jointly leverage appearance and motion cues, and a video-based Adaptive Multi-Prompts Refinement (AMPR) to select pivotal frames and form multi-prompts without extra training. The framework achieves state-of-the-art mIoU gains on MoCA-Mask and CAD benchmarks, while also delivering fast inference, demonstrating the effectiveness of integrating motion-appearance cues with SAM2 for robust VCOD. Overall, CamoSAM2 broadens SAM2’s applicability to camouflage-rich, real-world videos and offers a practical, automated solution for prompt generation and refinement.

Abstract

The Segment Anything Model 2 (SAM2), a prompt-guided video foundation model, has remarkably performed in video object segmentation, drawing significant attention in the community. Due to the high similarity between camouflaged objects and their surroundings, which makes them difficult to distinguish even by the human eye, the application of SAM2 for automated segmentation in real-world scenarios faces challenges in camouflage perception and reliable prompts generation. To address these issues, we propose CamoSAM2, a motion-appearance prompt inducer (MAPI) and refinement framework to automatically generate and refine prompts for SAM2, enabling high-quality automatic detection and segmentation in VCOD task. Initially, we introduce a prompt inducer that simultaneously integrates motion and appearance cues to detect camouflaged objects, delivering more accurate initial predictions than existing methods. Subsequently, we propose a video-based adaptive multi-prompts refinement (AMPR) strategy tailored for SAM2, aimed at mitigating prompt error in initial coarse masks and further producing good prompts. Specifically, we introduce a novel three-step process to generate reliable prompts by camouflaged object determination, pivotal prompting frame selection, and multi-prompts formation. Extensive experiments conducted on two benchmark datasets demonstrate that our proposed model, CamoSAM2, significantly outperforms existing state-of-the-art methods, achieving increases of 8.0% and 10.1% in mIoU metric. Additionally, our method achieves the fastest inference speed compared to current VCOD models.

Figures (5)

• Figure 1: Illustration of previous VCOD methods (a-b) with ours (c): (a) Feeding optical flow maps directly lamdouar2020betrayedyang2021selfsupervised; (b) Learning implicit motion cues from adjacent frames cheng2022implicit; (c) Learning motion-appearance guided prompts and subsequently refines these prompts automatically to enhance the effectiveness of SAM2.
• Figure 2: Pipeline of our CamoSAM2, which consists of two main components: motion-appearance prompt inducer (MAPI) and video-based adaptive multi-prompts refinement (AMPR). The fire and snowflake symbol signifies that the model parameters in this part are kept learnable and frozen, respectively.
• Figure 3: Structure of motion-guided appearance decoder.
• Figure 4: Visualization of our proposed CamoSAM2 and previous state-of-the-art methods on MoCA-Mask and CAD datasets.
• Figure 5: Visualization of our proposed adaptive multi-prompts refinement process. "Initial" represents the coarse masks after preprocessing in Step 1 of \ref{['alg:multi_prompt']}.