Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RenDetNet: Weakly-supervised Shadow Detection with Shadow Caster Verification

Nikolina Kubiak, Elliot Wortman, Armin Mustafa, Graeme Phillipson, Stephen Jolly, Simon Hadfield

TL;DR

The RenDetNet proposed in this paper is the first learning-based shadow detection model whose supervisory signals can be computed in a self-supervised manner and compares favourably against recent models trained on the data.

Abstract

Existing shadow detection models struggle to differentiate dark image areas from shadows. In this paper, we tackle this issue by verifying that all detected shadows are real, i.e. they have paired shadow casters. We perform this step in a physically-accurate manner by differentiably re-rendering the scene and observing the changes stemming from carving out estimated shadow casters. Thanks to this approach, the RenDetNet proposed in this paper is the first learning-based shadow detection model whose supervisory signals can be computed in a self-supervised manner. The developed system compares favourably against recent models trained on our data. As part of this publication, we release our code on github.

RenDetNet: Weakly-supervised Shadow Detection with Shadow Caster Verification

TL;DR

The RenDetNet proposed in this paper is the first learning-based shadow detection model whose supervisory signals can be computed in a self-supervised manner and compares favourably against recent models trained on the data.

Abstract

Existing shadow detection models struggle to differentiate dark image areas from shadows. In this paper, we tackle this issue by verifying that all detected shadows are real, i.e. they have paired shadow casters. We perform this step in a physically-accurate manner by differentiably re-rendering the scene and observing the changes stemming from carving out estimated shadow casters. Thanks to this approach, the RenDetNet proposed in this paper is the first learning-based shadow detection model whose supervisory signals can be computed in a self-supervised manner. The developed system compares favourably against recent models trained on our data. As part of this publication, we release our code on github.
Paper Structure (13 sections, 14 equations, 4 figures, 3 tables)

This paper contains 13 sections, 14 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Reducing the supervision requirements
  4. Summary
  5. Methodology
  6. Differentiable shadow caster verification with RenDetNet
  7. The RenDetNet approach to supervision
  8. Experiments
  9. The dataset
  10. Evaluation metrics and experimental setup
  11. Ablation study
  12. Comparisons with existing models
  13. Conclusions

Figures (4)

  • Figure 1: Examples of the common shadow vs dark region problem in shadow detection. The examples come from the ISTD dataset wang_2018_stacked and the estimated shadow masks (SMs) were generated using a SOTA shadow detection model chen2020multi. This shows that even some of the best available networks still struggle to distinguish between shadows and dark image regions, and inspires us to develop a detection model with improved scene understanding.
  • Figure 2: At train time, we take in lighting $\phi$, camera $\kappa$ and mesh $\mu$ parameters and render the scene $\textbf{I} = \mathcal{R}\left(\phi, \kappa, \mu\right)$. We push the render I through our RenDetNet to obtain the shadow mask SM and the caster mask CM. We then use CM to carve the mesh and re-render the scene, $\hat{\textbf{I}} = \mathcal{R}\left(\phi, \kappa, \hat{\mu}\right)$. At test time, only the region inside the dotted pink box is used.
  • Figure 3: Examples of RenDetNet's caster & shadow identification abilities - Dataset #2
  • Figure 4: Qualitative results -- row 1: Dataset #1; rows 2-3: Dataset #2.