Towards the Uncharted: Density-Descending Feature Perturbation for Semi-supervised Semantic Segmentation

TL;DR

The paper tackles label-efficient semantic segmentation by leveraging unlabeled data through feature-level consistency. It introduces Density-Descending Feature Perturbation (DDFP), which learns a real-time feature density $p_{\mathcal{V}}(v)$ via a lightweight normalizing-flow estimator and generates density-descending perturbations along the gradient of $-\log p_{\mathcal{V}}(v)$ to regularize the classifier. Under a teacher-student framework with image-level consistency and a unified objective $\mathcal{L}_{uni}$, DDFP achieves state-of-the-art results on Pascal VOC 2012 and Cityscapes across multiple data partitions. The approach offers strong generalization with minimal additional inference cost, since the density estimator is discarded at test time.

Abstract

Semi-supervised semantic segmentation allows model to mine effective supervision from unlabeled data to complement label-guided training. Recent research has primarily focused on consistency regularization techniques, exploring perturbation-invariant training at both the image and feature levels. In this work, we proposed a novel feature-level consistency learning framework named Density-Descending Feature Perturbation (DDFP). Inspired by the low-density separation assumption in semi-supervised learning, our key insight is that feature density can shed a light on the most promising direction for the segmentation classifier to explore, which is the regions with lower density. We propose to shift features with confident predictions towards lower-density regions by perturbation injection. The perturbed features are then supervised by the predictions on the original features, thereby compelling the classifier to explore less dense regions to effectively regularize the decision boundary. Central to our method is the estimation of feature density. To this end, we introduce a lightweight density estimator based on normalizing flow, allowing for efficient capture of the feature density distribution in an online manner. By extracting gradients from the density estimator, we can determine the direction towards less dense regions for each feature. The proposed DDFP outperforms other designs on feature-level perturbations and shows state of the art performances on both Pascal VOC and Cityscapes dataset under various partition protocols. The project is available at https://github.com/Gavinwxy/DDFP.

Figures (4)

• Figure 1: t-SNE visualization of per-pixel features from Pascal VOC 2012 dataset pascal. (a) Features extracted from encoder. (b) Features after the proposed DDFP strategy (shown in red). The perturbed features significantly deviate from high density centers and move towards low density regions within and out of clusters.
• Figure 2: Overview of the proposed density-descending feature perturbation strategy. Based on the weak-to-strong consistency regularization, our method consists of two main components: (a) Learning Density Distribution and (b) Perturbation Generation. In phase (a), a lightweight normalizing-flow-based estimator is adopted to learn the density information on unlabeled features from teacher encoder. A mixture of Gaussian distribution is initialized and the estimator is optimized to maximize the feature likelihood on that distribution, which is denoted by blue arrows. Meanwhile in phase (b), the density estimator is set as evaluation mode and applied on the student features. Once feature distribution is approximated, the density-descending direction can be extracted by the gradient of the density objective on unlabeled features. Predictions on the density-descending features are supervised by pseudo labels from teacher model. The whole perturbation generation and injection process is indicated by red arrows.
• Figure 3: Block design for the proposed density estimator.
• Figure 4: Qualitative results on Pascal VOC 2012 dataset. Models are trained under classic 732 partitions. (a) Input images. (b) Results from baseline model with image-level consistency regularization only. (c) Results by our DDFP. (d) Ground truth.