DTTNet: Improving Video Shadow Detection via Dark-Aware Guidance and Tokenized Temporal Modeling

TL;DR

Video shadow detection is challenged by shadow-background ambiguity and dynamic shadow deformations across time. DTTNet addresses this with a Vision-language Match Module that supplies text-guided priors and a Dark-aware Semantic Block that injects semantic guidance into encoder features, complemented by a Tokenized Temporal Block for efficient cross-frame temporal modeling. Penalized supervision for penumbra regions and edge-focused decoder supervision further stabilizes training. Experiments on ViSha and CVSD show state-of-the-art accuracy and real-time inference, highlighting the practical impact of combining vision-language priors with token-based temporal encoding for robust shadow localization in videos.

Abstract

Video shadow detection confronts two entwined difficulties: distinguishing shadows from complex backgrounds and modeling dynamic shadow deformations under varying illumination. To address shadow-background ambiguity, we leverage linguistic priors through the proposed Vision-language Match Module (VMM) and a Dark-aware Semantic Block (DSB), extracting text-guided features to explicitly differentiate shadows from dark objects. Furthermore, we introduce adaptive mask reweighting to downweight penumbra regions during training and apply edge masks at the final decoder stage for better supervision. For temporal modeling of variable shadow shapes, we propose a Tokenized Temporal Block (TTB) that decouples spatiotemporal learning. TTB summarizes cross-frame shadow semantics into learnable temporal tokens, enabling efficient sequence encoding with minimal computation overhead. Comprehensive Experiments on multiple benchmark datasets demonstrate state-of-the-art accuracy and real-time inference efficiency. Codes are available at https://github.com/city-cheng/DTTNet.

Figures (4)

• Figure 1: Comparison between existing methods and ours. Despite using intra-clip fusion, we utilize token to learn temporal characteristics and introduce textual priors to guide the model to learn from dark regions.
• Figure 2: Architecture of Dark-aware and Temporal Tokenized Network (DTTNet). DTTNet integrates dark-aware linguistic guidance with tokenized temporal modeling to effectively capture spatial and temporal dependencies. It consists of three novel modules: the Vision-language Match Module (VMM), Dark-aware Semantic Block and the Tokenized Temporal Block. We freeze most of the parameters and update only the parameter of decoder and proposed modules.
• Figure 3: Details of the proposed Dark-aware Semantic Block and Tokenized Temporal Block.
• Figure 4: Qualitative comparison results of state-of-the-art methods. In comparison to other methods, our results exhibit less noise, and the predictions for shadow boundaries are more accurate. (b-d) are the best methods in IOS, ISD, and VOS in Table \ref{['tab:res']}, and (e-g) are the latest networks in VSD.