Learning Local and Global Temporal Contexts for Video Semantic Segmentation

TL;DR

The paper tackles video semantic segmentation by explicitly modeling temporal context at two levels: local temporal contexts from nearby frames (static and motional) and global temporal contexts from the whole video. It introduces Coarse-to-Fine Feature Mining (CFFM) to jointly learn static and motional LTCs and Cross-frame Feature Mining (CFM) as a non-self attention mechanism to fuse context into target frame features. To extend temporal awareness to the full video, it adds CFFM++ which learns global temporal contextual prototypes via unsupervised clustering and refines target features through prototype-based mining. Experiments on VSPW and Cityscapes demonstrate state-of-the-art performance with modest computational overhead, validating the complementary benefits of LTC and GTC for robust, temporally consistent VSS.

Abstract

Contextual information plays a core role for video semantic segmentation (VSS). This paper summarizes contexts for VSS in two-fold: local temporal contexts (LTC) which define the contexts from neighboring frames, and global temporal contexts (GTC) which represent the contexts from the whole video. As for LTC, it includes static and motional contexts, corresponding to static and moving content in neighboring frames, respectively. Previously, both static and motional contexts have been studied. However, there is no research about simultaneously learning static and motional contexts (highly complementary). Hence, we propose a Coarse-to-Fine Feature Mining (CFFM) technique to learn a unified presentation of LTC. CFFM contains two parts: Coarse-to-Fine Feature Assembling (CFFA) and Cross-frame Feature Mining (CFM). CFFA abstracts static and motional contexts, and CFM mines useful information from nearby frames to enhance target features. To further exploit more temporal contexts, we propose CFFM++ by additionally learning GTC from the whole video. Specifically, we uniformly sample certain frames from the video and extract global contextual prototypes by k-means. The information within those prototypes is mined by CFM to refine target features. Experimental results on popular benchmarks demonstrate that CFFM and CFFM++ perform favorably against state-of-the-art methods. Our code is available at https://github.com/GuoleiSun/VSS-CFFM

Figures (4)

• Figure 1: Illustration of various video contexts. (a) Illustration of local temporal contexts and global temporal contexts. (b) Illustration of static contexts (in blue) and motional contexts (in red) across neighbouring video frames. The human and horse are moving objects, while the grassland and sky are static backgrounds. Note that the static stuff is helpful for the recognition of moving objects, i.e., a human is riding a horse on the grassland.
• Figure 2: Overview of the proposed Coarse-to-Fine Feature Mining for mining local temporal contexts. All frames are first input to an encoder to extract features, which then go through the coarse-to-fine feature assembling module (CFFA). Features for different frames are processed by different pooling strategies to generate the context tokens. The principle is that for more distant frames, a bigger receptive field and more coarse pooling are used. The shown feature size ($20 \times 20$), receptive field, and pooling kernel are for a simple explanation. The context tokens from all frames are concatenated and then processed by the cross-frame feature mining (CFM) module. The context tokens are exploited to update the target features by several multi-head non-self attention layers. Finally, we use the enhanced target features to make the segmentation prediction for the target frame. Best viewed with zooming.
• Figure 3: Overview of the proposed CFFM++ for additionally mining global temporal contexts. Due to the large number of frames in the video, we uniformly sample frames by a fixed step. The sampled video frames go through the encoder trained by CFFM and corresponding features are generated. After tokenizing the feature maps, we conduct unsupervised clustering ($k$-means) to reduce the tokens' number and learn global contextual prototypes. The obtained prototypes and the target frame features are passed to CFM, which enables the refinement of the target frame using global temporal contexts. The final predictions of CFFM++ are given by the weighted summation of the segmentation logits from learning local (CFFM) and global temporal contexts.
• Figure 4: Qualitative results for two video clips. We compare the proposed methods with the baseline (SegFormer xie2021segformer) visually. From top to down: the input video frames, the predictions of SegFormer xie2021segformer, CFFM predictions, CFFM++ predictions and the ground truth (GT). It shows that CFFM produces more accurate and consistent results, compared to the strong baseline. Furthermore, by using global temporal contexts, CFFM++ further improves over CFFM. Best viewed in color.