CM-MaskSD: Cross-Modality Masked Self-Distillation for Referring Image Segmentation

TL;DR

This work tackles referring image segmentation by addressing the persistent challenge of fine-grained cross-modality alignment between language and vision. It introduces CM-MaskSD, a cross-modality masked self-distillation framework that leverages CLIP-pretrained knowledge to guide dense patch-word correspondence, implemented via a main segmentation branch plus two symmetric masked self-distillation branches with shared weights for parameter efficiency. The approach enforces consistency between masked and unmasked segmentation outputs through two distillation pathways (LMVSD and VMLSD), yielding improved segmentation accuracy across RefCOCO, RefCOCO+, and G-Ref with negligible additional parameters. Extensive ablations validate the design choices, including correlation-guided masking, masking strategies, and weight-sharing, highlighting the method’s effectiveness and efficiency. The framework offers a plug-and-play solution for enhancing cross-modal alignment in RIS and potentially other multimodal vision-language tasks.

Abstract

Referring image segmentation (RIS) is a fundamental vision-language task that intends to segment a desired object from an image based on a given natural language expression. Due to the essentially distinct data properties between image and text, most of existing methods either introduce complex designs towards fine-grained vision-language alignment or lack required dense alignment, resulting in scalability issues or mis-segmentation problems such as over- or under-segmentation. To achieve effective and efficient fine-grained feature alignment in the RIS task, we explore the potential of masked multimodal modeling coupled with self-distillation and propose a novel cross-modality masked self-distillation framework named CM-MaskSD, in which our method inherits the transferred knowledge of image-text semantic alignment from CLIP model to realize fine-grained patch-word feature alignment for better segmentation accuracy. Moreover, our CM-MaskSD framework can considerably boost model performance in a nearly parameter-free manner, since it shares weights between the main segmentation branch and the introduced masked self-distillation branches, and solely introduces negligible parameters for coordinating the multimodal features. Comprehensive experiments on three benchmark datasets (i.e. RefCOCO, RefCOCO+, G-Ref) for the RIS task convincingly demonstrate the superiority of our proposed framework over previous state-of-the-art methods.

Figures (5)

• Figure 1: The illustration of our pipeline for referring image segmentation task.
• Figure 2: The architecture of our CM-MaskSD framework. It consists of a multimodal segmentation branch and two symmetric masked self-distillation branches that are designed for more fine-grained visual and textual feature alignment. During training, the main segmentation loss $Loss_{seg}$ coupled with two self-distillation loss $Loss_{LMVSD}$ and $Loss_{VMLSD}$ are jointly employed to pull close the segmentation masks generated by main branch and cross-modality guided masked self-distillation branches. For inference, only the main segmentation branch is preserved to acquire the final segmentation masks.
• Figure 3: The illustration of the introduced correlation filtering and cross-modality guided masking strategy in our language-guided masked visual self-distillation branch.
• Figure 4: The visual comparison of segmentation results on RefCOCO validation set. (a) input image. (b) CRIS. (c) our CM-MaskSD. (d) ground truth.
• Figure 5: Qualitative analysis for ablation study on our masked self-distillation design. (a) input image. (b) baseline. (c) our CM-MaskSD. (d) ground truth.