S$^3$POT: Contrast-Driven Face Occlusion Segmentation via Self-Supervised Prompt Learning

TL;DR

This work tackles occlusion misclassification in face parsing by introducing $S^3POT$, a contrast-driven, self-supervised framework that leverages a generated occlusion-free reference and SAM with learned prompts. The method unfolds through three modules—Reference Generation, Feature Enhancement, and Prompt Selection—that together convert input-reference contrast into effective occlusion prompts and a refined mask via a self-attention screening mechanism. It introduces three novel objective functions to supervise prompt selection without occlusion ground truth, achieving state-of-the-art Occlusion IoU on a dedicated occlusion dataset while maintaining facial geometry. The approach reduces annotation burden and is compatible with existing foundation-model pipelines, offering practical benefits for robust face analysis in occluded scenarios. Limitations remain for very fine-grained occlusions, pointing to future work in multi-scale fusion and uncertainty-aware masking.

Abstract

Existing face parsing methods usually misclassify occlusions as facial components. This is because occlusion is a high-level concept, it does not refer to a concrete category of object. Thus, constructing a real-world face dataset covering all categories of occlusion object is almost impossible and accurate mask annotation is labor-intensive. To deal with the problems, we present S$^3$POT, a contrast-driven framework synergizing face generation with self-supervised spatial prompting, to achieve occlusion segmentation. The framework is inspired by the insights: 1) Modern face generators' ability to realistically reconstruct occluded regions, creating an image that preserve facial geometry while eliminating occlusion, and 2) Foundation segmentation models' (e.g., SAM) capacity to extract precise mask when provided with appropriate prompts. In particular, S$^3$POT consists of three modules: Reference Generation (RF), Feature enhancement (FE), and Prompt Selection (PS). First, a reference image is produced by RF using structural guidance from parsed mask. Second, FE performs contrast of tokens between raw and reference images to obtain an initial prompt, then modifies image features with the prompt by cross-attention. Third, based on the enhanced features, PS constructs a set of positive and negative prompts and screens them with a self-attention network for a mask decoder. The network is learned under the guidance of three novel and complementary objective functions without occlusion ground truth mask involved. Extensive experiments on a dedicatedly collected dataset demonstrate S$^3$POT's superior performance and the effectiveness of each module.

Figures (10)

• Figure 1: Qualitative and quantitative results of SegFace trained on a large-scale synthetic dataset voo2022delving and tested on our collected data.
• Figure 2: Motivation and Idea Overview. (a) The illustration of the limitations of the current face-parsing algorithms when occlusions occur, which will lead to bad result in downstream tasks. (b) The demonstration of our brief idea and strategy of generating an appropriate prompt for SAM to accurately create an occlusion mask.
• Figure 3: The architecture of the proposed framework. The framework consists of three modules: RG, FE, and PS. RG: Given a face image $I$, a corresponding face parsing mask $M_r$ is first predicted and forwarded into a face generator $G$. The original image $I$ is used as a condition in $G$ for producing a reference face $I_r$. FE: Both $I_r$ and $I$ are fed into the SAM image encoder $E_I$ to obtain the image tokens $Z_r$ and $Z$, respectively. The cosine similarity between $Z_r$ and $Z$ is used to generate an initial prompt $p_i$. This prompt is first fed into prompt encoder $E_P$ and then paired separately with $Z_r$ and $Z$ before being processed by the feature adjustment component, resulting in enhanced tokens $Z_{re}$ and $Z_{e}$. The feature adjustment component incorporates multilayer perceptron (MLP), prompt-to-image cross-attention ($p_iZ$CA), and image-to-prompt cross-attention ($Zp_i$CA). PS: Using $M_r$, the facial region is selected from the enhanced tokens, producing $Z_{re}^{M_f}$ and $Z_{e}^{M_f}$. Then, the cosine similarity between $Z_{re}^{M_f}$ and $Z_{e}$ is computed and a greedy strategy is applied to find similar tokens in $Z_{e}^{M_f}$. Unpaired tokens in $Z_{e}^{M_f}$ are identified as occlusion prompts $P_O$, while the paired tokens constitute the non-occlusion prompts $P_N$. These prompts are processed together by the prompt encoder $E_P$ to obtain $P$. Following a self-attention screening layer (SA), the re-weighted prompt $P'$ is paired with $Z$ and forwarded to the SAM mask decoder $D_M$ to predict the occlusion mask $M_O$.
• Figure 4: Motivation for filtering redundancy and noise in the new prompt set $P$ . Even with a large number of accurate prompts input to SAM, performance does not improve. A smaller set of well-chosen, accurate prompts is sufficient for optimal results.
• Figure 5: The Illustration of the intuition of objective functions. The prompt points are categorized into two groups: occlusion points (black) and non-occlusion points (grey). In the predicted mask probability map, the probability values of occlusion points should be heightened, while those of non-occlusion points should be suppressed. When reducing the probabilities of non-occlusion points, in addition to lowering the average probability, we also introduce a penalty term to prevent individual points from exhibiting anomalously high values.