All-frequency Full-body Human Image Relighting

TL;DR

It is demonstrated that the proposed two‐stage relighting method can plausibly reproduce all‐frequency shadows and shading caused by environment illumination, which have been difficult to reproduce using existing methods.

Abstract

Relighting of human images enables post-photography editing of lighting effects in portraits. The current mainstream approach uses neural networks to approximate lighting effects without explicitly accounting for the principle of physical shading. As a result, it often has difficulty representing high-frequency shadows and shading. In this paper, we propose a two-stage relighting method that can reproduce physically-based shadows and shading from low to high frequencies. The key idea is to approximate an environment light source with a set of a fixed number of area light sources. The first stage employs supervised inverse rendering from a single image using neural networks and calculates physically-based shading. The second stage then calculates shadow for each area light and sums up to render the final image. We propose to make soft shadow mapping differentiable for the area-light approximation of environment lighting. We demonstrate that our method can plausibly reproduce all-frequency shadows and shading caused by environment illumination, which have been difficult to reproduce using existing methods.

Figures (17)

• Figure 1: Our method infers the materials (roughness, diffuse albedo, and specular) and geometry (depth and normal) from an input human image and calculates all-frequency shadows and reflections under new lighting conditions. As a lighting representation, we adopt a fixed number of area lights that approximate the target environment map.
• Figure 2: Overview of our two-stage relighting approach. Stage 1 applies inverse rendering and calculates a shadow-less shading image for each light. Stage 2 calculates shadows and multiplies shading images by the shadows and then merges them to output the relighting result.
• Figure 3: Validation of background-aware light estimation. Top row: spheres shaded with the estimated lights. Bottom row: estimated normal maps.
• Figure 4: Example data generated from a 3D human model.
• Figure 5: Overview of our area light optimization.