Leveraging Open-Vocabulary Diffusion to Camouflaged Instance Segmentation

TL;DR

This work tackles camouflaged instance segmentation in an open-vocabulary setting by integrating a frozen text-to-image diffusion model (Stable Diffusion) with CLIP-based textual guidance. The authors design camouflaged-specific components—multi-scale feature fusion (MSFF), a Mask2Former-based mask generator, textual-visual aggregation (TVA), and camouflaged instance normalization (CIN)—to learn textual-visual representations that distinguish camouflaged objects from clutter and generalize to unseen categories. Training combines BCE, Dice, and cross-entropy losses with Hungarian matching, while open-vocabulary inference enables segmentation of test categories without labels. Across CIS benchmarks COD10K-v3 and NC4K and open-vocabulary datasets ADE20K and Cityscapes, the method shows state-of-the-art or competitive performance with strong efficiency, and ablations confirm the critical role of the proposed modules and prompt design. This approach advances open-vocabulary CIS and holds promise for practical applications in wildlife monitoring, surveillance, and reconnaissance.

Abstract

Text-to-image diffusion techniques have shown exceptional capability of producing high-quality images from text descriptions. This indicates that there exists a strong correlation between the visual and textual domains. In addition, text-image discriminative models such as CLIP excel in image labelling from text prompts, thanks to the rich and diverse information available from open concepts. In this paper, we leverage these technical advances to solve a challenging problem in computer vision: camouflaged instance segmentation. Specifically, we propose a method built upon a state-of-the-art diffusion model, empowered by open-vocabulary to learn multi-scale textual-visual features for camouflaged object representations. Such cross-domain representations are desirable in segmenting camouflaged objects where visual cues are subtle to distinguish the objects from the background, especially in segmenting novel objects which are not seen in training. We also develop technically supportive components to effectively fuse cross-domain features and engage relevant features towards respective foreground objects. We validate our method and compare it with existing ones on several benchmark datasets of camouflaged instance segmentation and generic open-vocabulary instance segmentation. Experimental results confirm the advances of our method over existing ones. We will publish our code and pre-trained models to support future research.

Figures (9)

• Figure 1: Illustration of textual-visual features learnt by our method using text-to-image diffusion with open-vocabulary for CIS. Given an input image (1st and 3rd column), textual-visual features are extracted and clustered using a $K$-means clustering algorithm (2nd and 4th column). As shown, camouflaged animals (highlighted in red) can be localised from the clustering results. We leverage these rich features to perform instance segmentation of camouflaged objects. This figure is best viewed in colour.
• Figure 2: Pipeline of our proposed method for Camouflaged Instance Segmentation (CIS). Inputs include an image and a text prompt about target objects in the input image. Outputs include instance masks of the target objects. The target objects can be novel and have never been seen in the training data. We leverage state-of-the-art text-to-image diffusion and text-image transfer techniques to learn textual-visual features that facilitate the object representation learning for segmenting camouflaged objects.
• Figure 3: Architecture of the multi-scale features fusion (MSFF) module.
• Figure 4: Architecture of the mask generator.
• Figure 5: Architecture of the textual-visual aggregation (TVA) module.