SoftShadow: Leveraging Soft Masks for Penumbra-Aware Shadow Removal

TL;DR

SoftShadow tackles the challenge of removing soft shadows by introducing soft shadow masks predicted via a fine-tuned Segment Anything Model (SAM) with Low-Rank Adaptation (LoRA) and guiding a shadow-removal network with a penumbra-aware constraint. The framework jointly optimizes mask prediction and shadow removal using three losses: $L_{mask}$, $L_{pen}$, and $L_{rem}$, enabling end-to-end training without ground-truth masks. By modeling penumbra formation, the method achieves more accurate boundary handling and reduces artifacts, delivering state-of-the-art results on SRD and LRSS and strong generalization to UIUC and ISTD+. The approach presents practical impact by leveraging powerful external detectors (SAM) in a soft, differentiable manner to improve shadow removal in real-world images with nuanced illumination transitions.

Abstract

Recent advancements in deep learning have yielded promising results for the image shadow removal task. However, most existing methods rely on binary pre-generated shadow masks. The binary nature of such masks could potentially lead to artifacts near the boundary between shadow and non-shadow areas. In view of this, inspired by the physical model of shadow formation, we introduce novel soft shadow masks specifically designed for shadow removal. To achieve such soft masks, we propose a SoftShadow framework by leveraging the prior knowledge of pretrained SAM and integrating physical constraints. Specifically, we jointly tune the SAM and the subsequent shadow removal network using penumbra formation constraint loss, mask reconstruction loss, and shadow removal loss. This framework enables accurate predictions of penumbra (partially shaded) and umbra (fully shaded) areas while simultaneously facilitating end-to-end shadow removal. Through extensive experiments on popular datasets, we found that our SoftShadow framework, which generates soft masks, can better restore boundary artifacts, achieve state-of-the-art performance, and demonstrate superior generalizability.

Figures (8)

• Figure 1: Illustration of soft shadow removal results using our proposed SoftShadow with soft shadow mask compared to the recent competing methods BMNet zhu2022bijective, SG-ShadowNet wan2022style and HomoFormer xiao2024homoformer using hard shadow masks. The second rows are sharpened versions of results for better visualization.
• Figure 2: (a) Illustration of the shadow formation geometry that creates the penumbra and umbra nielsen2007segmentation regions; the umbra area is where the light is fully occluded, the penumbra area is where light is partially occluded, the lit area is where light is not occluded. (b) Examples of soft shadow images and hard shadow images from commonly used datasets.
• Figure 3: Illustration of the proposed SoftShadow. The left box illustrates the SoftShadow networks, where a shadow image $\mathbf{y}$ is input into SAM for detecting soft shadow masks. The shadow removal network then processes the soft mask and the shadow image to produce a shadow-free image. The right box shows the three losses we used in SoftShadow. From top to bottom, the shadow removal loss $\mathcal{L}_{rem}$ is calculated between shadow-free images and shadow-removal images. The mask reconstruction loss $\mathcal{L}_{mask}$ is calculated between predicted soft masks and ours ground truth soft masks. The penumbra formation constraint loss $\mathcal{L}_{pen}$ act as a regularization term. It aims to regularize the gradient of predicted soft masks in the penumbra area. As shown in the soft mask intensity curve, the ideal mask intensity in the penumbra area should not be too large and the gradient orientation should be consistence.
• Figure 4: Illustrate the concept of Penumbra Formation Constraint. The $w_0$ and $w_1$ are points in the inner boundary and the outer boundary of the penumbra area, respectively. In the ideal soft shadow mask, the intensity in the penumbra area decreases progressively from the shadow center to the umbra area. The gradient orientation points from the shadow center towards the non-shadow area.
• Figure 5: Examples of soft shadow image removal results on the SRD dataset qu2017deshadownet. The input shadow image, the estimated results of (a) BMNet zhu2022bijective, (b) ShadowDiffusion guo2023shadowdiffusion, (c) Inpaint4Shadow li2023leveraging, (d) Homoformer xiao2024homoformer, and (e) Ours, as well as the ground truth (GT) image, respectively.