Shadow Removal Refinement via Material-Consistent Shadow Edges

TL;DR

A new metric and an annotated dataset for evaluating the performance of shadow removal methods without the need for paired shadow/shadow-free data are proposed and demonstrated.

Abstract

Shadow boundaries can be confused with material boundaries as both exhibit sharp changes in luminance or contrast within a scene. However, shadows do not modify the intrinsic color or texture of surfaces. Therefore, on both sides of shadow edges traversing regions with the same material, the original color and textures should be the same if the shadow is removed properly. These shadow/shadow-free pairs are very useful but hard-to-collect supervision signals. The crucial contribution of this paper is to learn how to identify those shadow edges that traverse material-consistent regions and how to use them as self-supervision for shadow removal refinement during test time. To achieve this, we fine-tune SAM, an image segmentation foundation model, to produce a shadow-invariant segmentation and then extract material-consistent shadow edges by comparing the SAM segmentation with the shadow mask. Utilizing these shadow edges, we introduce color and texture-consistency losses to enhance the shadow removal process. We demonstrate the effectiveness of our method in improving shadow removal results on more challenging, in-the-wild images, outperforming the state-of-the-art shadow removal methods. Additionally, we propose a new metric and an annotated dataset for evaluating the performance of shadow removal methods without the need for paired shadow/shadow-free data.

Figures (22)

• Figure 1: Examples from our proposed SBU-S (top) and CUHK-S (bottom) testing sets. We show the shadow removal results of two existing state-of-the-art methods, SIDLe_2019_ICCV and ShadowFormerguo2023shadowformer in columns (b) and (d) respectively for two challenging cases. The results of both methods are significantly improved when used jointly with our refinement method - as in columns (c) and (e). Our proposed Color Distribution Difference (CDD) metric for each image is shown in red, which can measure shadow removal performance without the need for shadow-free images.
• Figure 1: More examples of improved material-consistent mask segmentation by our fine-tuned SAM.
• Figure 2: Fine-tuning SAM for Material-Consistent Edge Extraction. Given the input image, we compare the segmentation results of vanilla SAM kirillov2023segany with our fine-tuned SAM. Our fine-tuned SAM achieves shadow-invariant segmentation, preserving the material consistency of each mask. In contrast, vanilla SAM is sensitive to shadow presence, segmenting shadow regions as individual masks.
• Figure 2: Correctness of Color Distribution Difference metric. (top) We manually adjust the shadow intensity from strong to weak, and the CDD values are lower when the shadow effect is weaker. (bottom) We compare the CDD values of shadow images and their shadow-free counterparts, the CDD value of the shadow-free version is at least two orders of magnitude lower than the shadow version. CDD are computed using the pixels marked in the images and the values are reported in the images.
• Figure 3: Supervision for the adaptation. Given the input image and the shadow mask, we first use the fine-tuned SAM to produce material-consistent (MC) masks. We sample pixels on both sides alongside the MC shadow edge, denoted as $S_{in}$ (shown in red) and $S_{out}$ (shown in green), and patches within the same material, denoted as $P_{in}$ (shown in red) and $P_{out}$ (shown in green). Pixels and patches on both sides of the shadow edge provide supervision for the following adaptation process.