Shift-Window Meets Dual Attention: A Multi-Model Architecture for Specular Highlight Removal

TL;DR

This work tackles specular highlight removal in single images by proposing MM-SHR, a hybrid CNN–Transformer architecture that jointly captures fine local details and global dependencies across scales. It introduces three novel components: OAIBlock for omnidirectional, frequency-guided attention; HDDAConv for region-aware dual-attention in a hybrid domain; and HDCTransformer for efficient global modeling at downsampled resolutions, aided by frequency-domain guidance. A composite loss combining pixel, structural, and perceptual terms with a dual-output design enforces proper separation of diffuse content and specular residue. Extensive quantitative and qualitative evaluations on three benchmark datasets show that MM-SHR achieves state-of-the-art accuracy with lower inference cost than transformer-heavy counterparts, indicating strong practical impact for real-time vision tasks in challenging lighting conditions.

Abstract

Inevitable specular highlights in practical environments severely impair the visual performance, thus degrading the task effectiveness and efficiency. Although there exist considerable methods that focus on local information from convolutional neural network models or global information from transformer models, the single-type model falls into a modeling dilemma between local fine-grained details and global long-range dependencies, thus deteriorating for specular highlights with different scales. Therefore, to accommodate specular highlights of all scales, we propose a multi-model architecture for specular highlight removal (MM-SHR) that effectively captures fine-grained features in highlight regions and models long-range dependencies between highlight and highlight-free areas. Specifically, we employ convolution operations to extract local details in the shallow layers of MM-SHR, and utilize the attention mechanism to capture global features in the deep layers, ensuring both operation efficiency and removal accuracy. To model long-range dependencies without compromising computational complexity, we utilize a coarse-to-fine manner and propose Omni-Directional Attention Integration Block(OAIBlock) and Adaptive Region-Aware Hybrid-Domain Dual Attention Convolutional Network(HDDAConv) , which leverage omni-directiona pixel-shifting and window-dividing operations at the raw features to achieve specular highlight removal. Extensive experimental results on three benchmark tasks and six types of surface materials demonstrate that MM-SHR outperforms state-of-the-art methods in both accuracy and efficiency for specular highlight removal. The implementation will be made publicly available at https://github.com/Htcicv/MM-SHR.

Figures (4)

• Figure 1: Overall architecture of Shift-Window Meets Dual Attention: A Multi-Model Architecture for Specular Highlight Removal
• Figure 2: Diagram of the Shift-Window process
• Figure 3: Qualitative comparison of MM-SHR with leading state-of-the-art highlight removal methods.
• Figure 4: Zero-shot test results demonstrating the generalization capability of MM-SHR.