Context Patch Fusion With Class Token Enhancement for Weakly Supervised Semantic Segmentation

TL;DR

This work tackles weakly supervised semantic segmentation by addressing the limitations of CAM-based localization through explicit patch-context modeling and class-level semantic refinement. It introduces CPF-CTE, which combines a Contextual-Fusion Bidirectional LSTM (CF-BiLSTM) for spatial patch dependencies with learnable class tokens applied post-ViT encoding to provide dynamic class-specific refinement, all integrated into a ViT-based framework. The model employs Top-$k$ pooling, a lightweight patch classifier, CRF refinement, and a DeepLabv2 refinement stage to produce high-quality pseudo-labels that supervise segmentation; it achieves state-of-the-art results on VOC-2012 and COCO-2014, with ablations confirming the contributions of both the CF-BiLSTM and class-token components. Overall, CPF-CTE offers a scalable, interpretable approach that improves intra-image contextual reasoning and class discrimination under weak supervision, enabling more accurate and robust semantic segmentation.

Abstract

Weakly Supervised Semantic Segmentation (WSSS), which relies only on image-level labels, has attracted significant attention for its cost-effectiveness and scalability. Existing methods mainly enhance inter-class distinctions and employ data augmentation to mitigate semantic ambiguity and reduce spurious activations. However, they often neglect the complex contextual dependencies among image patches, resulting in incomplete local representations and limited segmentation accuracy. To address these issues, we propose the Context Patch Fusion with Class Token Enhancement (CPF-CTE) framework, which exploits contextual relations among patches to enrich feature representations and improve segmentation. At its core, the Contextual-Fusion Bidirectional Long Short-Term Memory (CF-BiLSTM) module captures spatial dependencies between patches and enables bidirectional information flow, yielding a more comprehensive understanding of spatial correlations. This strengthens feature learning and segmentation robustness. Moreover, we introduce learnable class tokens that dynamically encode and refine class-specific semantics, enhancing discriminative capability. By effectively integrating spatial and semantic cues, CPF-CTE produces richer and more accurate representations of image content. Extensive experiments on PASCAL VOC 2012 and MS COCO 2014 validate that CPF-CTE consistently surpasses prior WSSS methods.

Figures (7)

• Figure 1: Comparison between traditional WSSS pipeline and the proposed CPF-CTE framework. (a) Traditional WSSS relies solely on ViT patch embeddings, which often suffer from fragmented and class-ambiguous activations. (b) Our CPF-CTE explicitly incorporates image-level class semantics through learnable class tokens and enhances spatial continuity using the Context Fusion module, leading to more coherent and discriminative feature representations for pseudo-label generation.
• Figure 2: Overview of the proposed CPF-CTE framework. Given an input RGB image, a ViT encoder extracts patch-level features $\mathbf{F}^{\text{vit}}$, which are then concatenated to form the input token sequence $\mathbf{F}^{\text{in}}$. Class information is encoded into learnable class tokens that are processed in parallel. The CF-BiLSTM module performs contextual fusion through both vertical (BiLSTM$_V$) and horizontal (BiLSTM$_H$) bidirectional propagation, producing refined representations $\mathbf{F}^{\text{out}}$. A patch classifier (MLP$_W$ followed by a sigmoid function) generates patch-level activation maps. These predictions are further aggregated with the image-level classifier and optimized using the multi-class entropy (MCE) loss.
• Figure 3: Pipeline of pseudo-label refinement and final segmentation prediction. The patch classifier outputs a baseline pseudo mask (BPM), which is refined by CRF to obtain cleaner pseudo labels. These refined pseudo labels are then used to train a DeepLabv2 segmentation network, whose output forms the final prediction.
• Figure 4: The impact of different K values on the final performance of WSSS.
• Figure 5: The impact of different patch values on the final performance of WSSS.