Language-Driven Interactive Shadow Detection

TL;DR

This work addresses the need to segment specific shadows in videos using natural language prompts, introducing the Referring Video Shadow Detection (RVSD) task and the RVSD dataset (86 videos, 15,011 text–shadow pairs). It proposes the Referring Shadow-Track Memory Network (RSM-Net), which combines Twin-Track Synergistic Memory (TSM) to store intra- and inter-clip features with a Mixed-Prior Shadow Attention (MSA) to generate a coarse shadow prior to refinement. The approach achieves state-of-the-art performance on RVSD, with a notable Overall IoU improvement of 4.4% over competing methods, validated on the new dataset. This work enables interactive video editing and immersive content creation by enabling language-driven, per-shadow segmentation in videos.

Abstract

Traditional shadow detectors often identify all shadow regions of static images or video sequences. This work presents the Referring Video Shadow Detection (RVSD), which is an innovative task that rejuvenates the classic paradigm by facilitating the segmentation of particular shadows in videos based on descriptive natural language prompts. This novel RVSD not only achieves segmentation of arbitrary shadow areas of interest based on descriptions (flexibility) but also allows users to interact with visual content more directly and naturally by using natural language prompts (interactivity), paving the way for abundant applications ranging from advanced video editing to virtual reality experiences. To pioneer the RVSD research, we curated a well-annotated RVSD dataset, which encompasses 86 videos and a rich set of 15,011 paired textual descriptions with corresponding shadows. To the best of our knowledge, this dataset is the first one for addressing RVSD. Based on this dataset, we propose a Referring Shadow-Track Memory Network (RSM-Net) for addressing the RVSD task. In our RSM-Net, we devise a Twin-Track Synergistic Memory (TSM) to store intra-clip memory features and hierarchical inter-clip memory features, and then pass these memory features into a memory read module to refine features of the current video frame for referring shadow detection. We also develop a Mixed-Prior Shadow Attention (MSA) to utilize physical priors to obtain a coarse shadow map for learning more visual features by weighting it with the input video frame. Experimental results show that our RSM-Net achieves state-of-the-art performance for RVSD with a notable Overall IOU increase of 4.4\%. Our code and dataset are available at https://github.com/whq-xxh/RVSD.

Figures (7)

• Figure 1: Comparisons of task settings for video shadow detection, instance shadow detection, and our RVSD. Traditional shadow detection (a) segments all shadows, current instance shadow detection (b) detects the foreground objects and segments the associated shadows, while our RVSD (c) can flexibly segment any shadow of interest referred by the text description, including I. those of multiple objects and II. background shadows (objects are invisible in the figure).
• Figure 2: Sample frames from the RVSD dataset showcase pixel-level shadow annotations paired with textual descriptions that guide the corresponding shadow segmentation. These examples demonstrate that our RVSD not only facilitates the flexible segmentation of a specific shadow but also effectively segments shadows cast by groups of objects.
• Figure 3: Word cloud of the RVSD dataset. The RVSD dataset encompasses a vast vocabulary that captures shadows from various perspectives, encompassing aspects like shadow type, location, shape, movement, and associated objects.
• Figure 4: An overview of our approach. The TSM on the left side represents the construction phase of twin-track memory, which contains both inter-clip and intra-clip tracks of memory. The clip block (intra-clip) in the lower left corner signifies memory propagation between frames, with each clip in the figure containing three video frames (Language and Image Integration). Eventually, The hierarchical memory is strategically accessed for processing the current frame, ensuring comprehensive and context-aware shadow detection.
• Figure 5: Details of hierarchical memory reading. First, perform memory embedding on it and then input it into the self-attention module together with the $\textbf{Learnable\ query}$.