Prompt-Aware Controllable Shadow Removal

TL;DR

This work tackles the limitation of existing shadow removal methods by enabling prompt-based controllable removal of shadows from user-specified subjects without requiring shadow region annotations. It introduces PACSRNet, a two-branch architecture comprising a prompt-aware module that generates subject-specific shadow masks and guidance via a spatial-frequency interaction, and a shadow removal module that uses dense-sparse local attention to reconstruct shadowed content guided by the prompt. A novel PCSRD dataset with 11,900 samples across dot, line, and subject-mask prompts is proposed to reflect realistic user interactions. Experimental results on PCSRD and ISTD+ demonstrate competitive or superior restoration quality compared to state-of-the-art baselines, with ablations validating the contributions of SFI, DSLA, and prompt-guided guidance. Overall, the approach provides flexible, annotation-free shadow editing suitable for practical applications in image editing and vision tasks requiring robust shadow removal.

Abstract

Shadow removal aims to restore the image content in shadowed regions. While deep learning-based methods have shown promising results, they still face key challenges: 1) uncontrolled removal of all shadows, or 2) controllable removal but heavily relies on precise shadow region masks. To address these issues, we introduce a novel paradigm: prompt-aware controllable shadow removal. Unlike existing approaches, our paradigm allows for targeted shadow removal from specific subjects based on user prompts (e.g., dots, lines, or subject masks). This approach eliminates the need for shadow annotations and offers flexible, user-controlled shadow removal. Specifically, we propose an end-to-end learnable model, the Prompt-Aware Controllable Shadow Removal Network (PACSRNet). PACSRNet consists of two key modules: a prompt-aware module that generates shadow masks for the specified subject based on the user prompt, and a shadow removal module that uses the shadow prior from the first module to restore the content in the shadowed regions. Additionally, we enhance the shadow removal module by incorporating feature information from the prompt-aware module through a linear operation, providing prompt-guided support for shadow removal. Recognizing that existing shadow removal datasets lack diverse user prompts, we contribute a new dataset specifically designed for prompt-based controllable shadow removal. Extensive experimental results demonstrate the effectiveness and superiority of PACSRNet.

Figures (7)

• Figure 1: Comparison with existing shadow removal methods. (a) Shadow unaware removes all the shadow regions with only raw images as input. (b) Shadow aware removes the shadow regions corresponding to the given shadow mask. (c) Our prompt-aware removal. Different from (a) and (b), the proposed method allows the removal of any subject's shadow with various prompts (i.e., dot, line, and subject mask).
• Figure 2: The overview of the our PACSRNet. PACSRNet is composed of a prompt-aware module and a shadow removal module. Prompt-aware module takes a shadow image and a prompt $\textbf{c}$ as inputs generating corresponding shadow mask $\widehat{\textbf{m}}$ and prompt-aware guidance feature. The generated shadow mask will be served as explicit guidance and fed into the shadow removal module along with the shadow image for shadow removal. The prompt-aware guidance feature is applied to encoder of shadow removal module guiding the shadow removal implicitly.
• Figure 3: Examples of shadow removal results based on dot, line and subject mask prompts. In the same image, we use the prompt to specify the different subjects and perform corresponding controllable shadow removal.
• Figure 4: Visualization of prompt-aware guidance feature maps. It implicitly guides the shadow removal module to focus on shadow regions marked in red dashed box.
• Figure 5: Visual comparison of the predicted shadow region masks for one subject under three different prompts. Our PACSRNet is robust to different types of user prompts.