MVGD-Net: A Novel Motion-aware Video Glass Surface Detection Network

TL;DR

This paper tackles video-based glass surface detection by leveraging motion inconsistency cues observed in optical-flow maps. It introduces MVGD-Net, a network with Cross-scale Multimodal Fusion Module (CMFM), History Guided Attention Module (HGAM), Temporal Cross Attention Module (TCAM), and Temporal-Spatial Decoder (TSD) to fuse spatial and temporal information. A large-scale MVGD-D dataset with 312 videos (19,268 frames) and manual glass masks is released to support VGSD research. Experiments show MVGD-Net achieves state-of-the-art performance on MVGD-D and VGSD-D, demonstrating the effectiveness of motion cues for robust glass-surface detection in real-world videos.

Abstract

Glass surface ubiquitous in both daily life and professional environments presents a potential threat to vision-based systems, such as robot and drone navigation. To solve this challenge, most recent studies have shown significant interest in Video Glass Surface Detection (VGSD). We observe that objects in the reflection (or transmission) layer appear farther from the glass surfaces. Consequently, in video motion scenarios, the notable reflected (or transmitted) objects on the glass surface move slower than objects in non-glass regions within the same spatial plane, and this motion inconsistency can effectively reveal the presence of glass surfaces. Based on this observation, we propose a novel network, named MVGD-Net, for detecting glass surfaces in videos by leveraging motion inconsistency cues. Our MVGD-Net features three novel modules: the Cross-scale Multimodal Fusion Module (CMFM) that integrates extracted spatial features and estimated optical flow maps, the History Guided Attention Module (HGAM) and Temporal Cross Attention Module (TCAM), both of which further enhances temporal features. A Temporal-Spatial Decoder (TSD) is also introduced to fuse the spatial and temporal features for generating the glass region mask. Furthermore, for learning our network, we also propose a large-scale dataset, which comprises 312 diverse glass scenarios with a total of 19,268 frames. Extensive experiments demonstrate that our MVGD-Net outperforms relevant state-of-the-art methods.

Figures (11)

• Figure 1: Examples of the motion inconsistency in indoor and outdoor scenes. The $1st$ and $2nd$ columns show the frames at time $t$ and time $t+10$, respectively. The $3rd$ column shows the inconsistent motion cues on glass surfaces, indicated by arrows. The $4th$ column shows the optical flow maps.
• Figure 2: Existing methods may under/over-detect glass surfaces in challenge scenes. Our method utilizes motion inconsistency cues to guide GSD and outperforms competitors.
• Figure 3: The structure of our proposed Motion-aware Video Glass Surface Detection Network (MVGD-Net).
• Figure 4: The structure of our CMFM.
• Figure 5: The structure of our HGAM.