Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes

TL;DR

This work tackles the challenge of high-cost cross-frame reasoning in video semantic segmentation for driving scenes. It introduces vanishing-point priors as a dual static-dynamic cue and builds VPSeg with two novel modules: MotionVP for VP-guided cross-frame motion estimation and DenseVP for VP-region scale-adaptive feature refinement. These modules operate in a context-detail framework and are fused with high-resolution local features through Contextualized Motion Attention (CMA), yielding accurate predictions with modest overhead. Empirical results on Cityscapes and ACDC demonstrate state-of-the-art performance, validating the practicality and effectiveness of leveraging vanishing points to guide video segmentation in real-world driving scenarios.

Abstract

The estimation of implicit cross-frame correspondences and the high computational cost have long been major challenges in video semantic segmentation (VSS) for driving scenes. Prior works utilize keyframes, feature propagation, or cross-frame attention to address these issues. By contrast, we are the first to harness vanishing point (VP) priors for more effective segmentation. Intuitively, objects near VPs (i.e., away from the vehicle) are less discernible. Moreover, they tend to move radially away from the VP over time in the usual case of a forward-facing camera, a straight road, and linear forward motion of the vehicle. Our novel, efficient network for VSS, named VPSeg, incorporates two modules that utilize exactly this pair of static and dynamic VP priors: sparse-to-dense feature mining (DenseVP) and VP-guided motion fusion (MotionVP). MotionVP employs VP-guided motion estimation to establish explicit correspondences across frames and help attend to the most relevant features from neighboring frames, while DenseVP enhances weak dynamic features in distant regions around VPs. These modules operate within a context-detail framework, which separates contextual features from high-resolution local features at different input resolutions to reduce computational costs. Contextual and local features are integrated through contextualized motion attention (CMA) for the final prediction. Extensive experiments on two popular driving segmentation benchmarks, Cityscapes and ACDC, demonstrate that VPSeg outperforms previous SOTA methods, with only modest computational overhead.

Figures (8)

• Figure 1: Illustration of the intuition behind our proposed vanishing-point-guided motion estimation and scale-adaptive partition modules. Targets move radially away from the vanishing point as time progresses in the video for the typical case of a forward-facing camera, a straight road and linear forward motion, which is depicted in this example. Moreover, the region around the vanishing point contains more distant objects, which appear at smaller scales.
• Figure 2: Overview of our VPSeg network. In the MotionVP module (bottom part), video frames are downsampled to extract context features, which go through cross-attention to capture dynamic context ${F}'_{t}$. ${F}'_{t}$ is further augmented by DenseVP to mine finer features ${f}_{\mathbf{A}}$ in the VP region through a two-scale partition strategy. In the upper part, we obtain local context ${F}_{tl}$ and local details ${F}_{th}$ from downsampled and high-resolution target frames $I_t$, respectively. In CMA, augmented dynamic context ${F}"_{t}$ interacts with local context ${F}_{tl}$ to generate the detail attention map $O$, guiding its merging with high-resolution local details ${F}_{th}$ for the final prediction ${P}_{f}$. Zoomed in for best view.
• Figure 3: Visualization of detail attention maps $O$ with $N$ motion attention layers in CMA. As $N$ increases, the detail attention map interacts more heavily with the dynamic features, and the weights gradually decrease in closer parts of the scene or on simple semantic categories. The highlighted distant regions near the VP suggest that the final predictions $P_f$ are primarily based on the detail-based predictions $P_{d}$ and not on $P_c$ for these regions. The VP proximity map serves as a positional prior and assists the model in pinpointing the locations of these distant regions.
• Figure 4: Qualitative comparison on ACDC. The yellow box represents the densely partitioned VP region. Our model produces more accurate results for both distant tiny hard samples near the VP and occluded fast-moving close targets.
• Figure 5: The VP detection pipeline. We first pre-process the input frame with morphology opening transform morph and Canny edge filtering canny. Hough-transform hough1 is then applied and lines that do not contribute to VP detection are discarded. Finally, cell vote is implemented to count the intersections in each cell to determine the final VP position.