Time2General: Learning Spatiotemporal Invariant Representations for Domain-Generalization Video Semantic Segmentation

TL;DR

Time2General addresses domain-generalized video semantic segmentation (DGVSS) under domain and temporal-sampling shifts by freezing a powerful backbone and introducing Stability Queries as temporally persistent semantic anchors. A Spatio-Temporal Memory Decoder aggregates multi-frame context without explicit correspondence, while a Masked Temporal Consistency Loss regularizes predictions across varying temporal strides. The approach achieves improved cross-domain accuracy and temporal stability across multiple driving benchmarks, running in real time (approximately 18 FPS) on standard hardware. These innovations enable more robust, flicker-resistant video segmentation suitable for real-world deployment without test-time adaptation.

Abstract

Domain Generalized Video Semantic Segmentation (DGVSS) is trained on a single labeled driving domain and is directly deployed on unseen domains without target labels and test-time adaptation while maintaining temporally consistent predictions over video streams. In practice, both domain shift and temporal-sampling shift break correspondence-based propagation and fixed-stride temporal aggregation, causing severe frame-to-frame flicker even in label-stable regions. We propose Time2General, a DGVSS framework built on Stability Queries. Time2General introduces a Spatio-Temporal Memory Decoder that aggregates multi-frame context into a clip-level spatio-temporal memory and decodes temporally consistent per-frame masks without explicit correspondence propagation. To further suppress flicker and improve robustness to varying sampling rates, the Masked Temporal Consistency Loss is proposed to regularize temporal prediction discrepancies across different strides, and randomize training strides to expose the model to diverse temporal gaps. Extensive experiments on multiple driving benchmarks show that Time2General achieves a substantial improvement in cross-domain accuracy and temporal stability over prior DGSS and VSS baselines while running at up to 18 FPS. Code will be released after the review process.

Figures (7)

• Figure 1: We visualize semantic segmentation results on consecutive frames under multiple weather domains. Existing methods show a clear drop in accuracy in unseen weather, with drifting object boundaries, label switching, and poor temporal coherence. In contrast, our method produces more consistent predictions across time under domain shift.
• Figure 2: Overview of the proposed Time2General framework with Stability Queries and Spatio-Temporal Memory Decoder for cross-frame consistent DGVSS.
• Figure 3: Qualitative comparison of cross domain segmentation predictions for KITTI-360 $\rightarrow$ City.-s + City.-s-C.
• Figure 4: Qualitative comparison of KITTI-360 $\rightarrow$ CamVid, Apollo. and Apollo. $\rightarrow$ CamVid, KITTI-360 predictions.
• Figure 5: Effect of Trimming Ratio on MTC Loss.