DOEI: Dual Optimization of Embedding Information for Attention-Enhanced Class Activation Maps

TL;DR

DOEI tackles the CAM quality bottleneck in weakly supervised semantic segmentation by aligning activation responses with semantic structure in high-dimensional embedding space. It introduces Dual Optimization of Embedding Information (DOEI), consisting of Patch-wise Progressive Decoupling Optimization (PPDO) and Class-wise Progressive Decoupling Optimization (CPDO), along with a Hybrid Feature Alignment (HFA) module to fuse RGB, embedding similarity, and self-attention signals. Integrated into ViT-based WSSS models, DOEI improves seed maps, pseudo-label quality, and final segmentation accuracy on PASCAL VOC and MS COCO, with VOC gains of up to +3.6/ +1.5/ +1.2 mIoU and COCO gains of +1.2/+1.6 mIoU reported; the method can be applied without introducing additional learnable parameters. Overall, DOEI demonstrates that coupling attention-driven embedding updates with multimodal feature fusion yields more complete object localization and robust segmentation in weakly supervised settings, advancing practical WSSS for complex scenes.

Abstract

Weakly supervised semantic segmentation (WSSS) typically utilizes limited semantic annotations to obtain initial Class Activation Maps (CAMs). However, due to the inadequate coupling between class activation responses and semantic information in high-dimensional space, the CAM is prone to object co-occurrence or under-activation, resulting in inferior recognition accuracy. To tackle this issue, we propose DOEI, Dual Optimization of Embedding Information, a novel approach that reconstructs embedding representations through semantic-aware attention weight matrices to optimize the expression capability of embedding information. Specifically, DOEI amplifies tokens with high confidence and suppresses those with low confidence during the class-to-patch interaction. This alignment of activation responses with semantic information strengthens the propagation and decoupling of target features, enabling the generated embeddings to more accurately represent target features in high-level semantic space. In addition, we propose a hybrid-feature alignment module in DOEI that combines RGB values, embedding-guided features, and self-attention weights to increase the reliability of candidate tokens. Comprehensive experiments show that DOEI is an effective plug-and-play module that empowers state-of-the-art visual transformer-based WSSS models to significantly improve the quality of CAMs and segmentation performance on popular benchmarks, including PASCAL VOC (+3.6%, +1.5%, +1.2% mIoU) and MS COCO (+1.2%, +1.6% mIoU). Code will be available at https://github.com/AIGeeksGroup/DOEI.

Figures (7)

• Figure 1: Visualization comparison between the baseline (MCTformer xu2022multi) and our method. The baseline model results exhibit background noise (object co-occurrence) and less precise localization (under-activation), as highlighted by red arrows and white bounding boxes. In contrast, our proposed method effectively reduces background activation and enhances focus on target regions, as highlighted by yellow arrows.
• Figure 1: Impact of varying numbers of embedding optimization layers on the mIoU, FP, and FN of the final CAM.
• Figure 2: (a) The image and query targets ($\star$). (b) The self-attention maps in the Transformer block capture semantic-level relationships at various granularities. The high-activation regions learned by each layer not only provide critical information that subsequent layers may miss but also generate CAMs that often focus on different regions. This broadens the coverage of target feature areas, effectively mitigating the tendency of activation maps to focus excessively on local salient regions.
• Figure 2: Visualization of the CAMs of input images generated by different methods. (a)(e) Input; (b)(f) Baseline; (c)(g) Dual Optimization of Embedding Information (DOEI); (d)(h) GT;
• Figure 3: An overview of the proposed method. The RGB image undergoes transformation into class tokens and patch tokens via the embedding layer. The dot-product attention mechanism is then employed to compute the attention score matrix and generate output tokens. To further refine the attention score matrix, the cosine similarity between the RGB values of the original image and the tokens is used to adjust the distribution of attention weights. Furthermore, the CPDO and PPDO methods are customized to amplify high-confidence information and suppress the influence of low-confidence information. Finally, the optimized tokens are incorporated into the original tokens as residuals, producing refined output tokens for subsequent computations in the encoder.