OmniSR: Shadow Removal under Direct and Indirect Lighting

TL;DR

OmniSR addresses shadow removal under both direct and indirect illumination by introducing a path-traced INdirect Shadow (INS) dataset and a RGBD-based semantic-geometry aware network. The model uses Depth-Anything-V2 for depth, DINO features for semantic cues, and Context-aware Swin Attention with semantic and geometric weighting to jointly remove direct and indirect shadows. Empirical results on INS and real-world data surpass state-of-the-art methods, demonstrating robust generalization to complex lighting in indoor and outdoor scenes. This work improves practical shadow removal for downstream vision tasks in realistic environments by providing high-quality training data and a powerful, geometry-informed network.

Abstract

Shadows can originate from occlusions in both direct and indirect illumination. Although most current shadow removal research focuses on shadows caused by direct illumination, shadows from indirect illumination are often just as pervasive, particularly in indoor scenes. A significant challenge in removing shadows from indirect illumination is obtaining shadow-free images to train the shadow removal network. To overcome this challenge, we propose a novel rendering pipeline for generating shadowed and shadow-free images under direct and indirect illumination, and create a comprehensive synthetic dataset that contains over 30,000 image pairs, covering various object types and lighting conditions. We also propose an innovative shadow removal network that explicitly integrates semantic and geometric priors through concatenation and attention mechanisms. The experiments show that our method outperforms state-of-the-art shadow removal techniques and can effectively generalize to indoor and outdoor scenes under various lighting conditions, enhancing the overall effectiveness and applicability of shadow removal methods.

Figures (6)

• Figure 1: Top: Removing shadows from intricate indoor scenes with indirect illumination poses a challenge for current methods such as ShadowFormer guo2023shadowformer and DMTN liu2023decoupled, especially when trained on existing datasets focusing on direct illumination, like ISTD+ le2019shadow. In contrast, our shadow removal network, along with a newly introduced INdirect Shadow (INS) dataset, demonstrates a superior ability to remove shadows accurately. Bottom: The rendering results reveal the prominence and prevalence of both direct and indirect shadows in indoor scenes.
• Figure 2: Rendering of shadow and shadow-free images.Left: The rendered image with shadow is a combination of a direct illumination image and an indirect illumination image. Right: The final shadow-free image is a composite of a direct shadow-free image, and an indirect shadow-free image.
• Figure 3: Examples of our proposed dataset. The rendered shadow and shadow-free image pairs and the generated shadow mask.
• Figure 4: Our proposed network. For each input RGB image, we first extract its DINO features, depth, and normal map. Then, the RGB and depth images are inputted into the shadow removal network. The network includes several Context-aware Swin Attention (CSA) layers, each comprising two Swin self-attention blocks. Unlike traditional self-attention, our block explicitly involves semantic-aware and geometry-aware attention weights.
• Figure 5: Comparisons with SOTA shadow removal methods show improved quality of our method. Comparisons with DMTN liu2023decoupled, ShadowFormer guo2023shadowformer, ShadowDiffusion guo2023shadowdiffusion, BMNet zhu2022bijective and Fu et al. fu2021auto on both outdoor and indoor scenes. Our method demonstrates more comprehensive shadow removal, even in complex scenes.