COMPOSE: Comprehensive Portrait Shadow Editing

TL;DR

This paper introduces COMPOSE: a novel shadow editing pipeline for human portraits, offering precise control over shadow attributes such as shape, intensity, and position, all while preserving the original environmental illumination of the portrait.

Abstract

Existing portrait relighting methods struggle with precise control over facial shadows, particularly when faced with challenges such as handling hard shadows from directional light sources or adjusting shadows while remaining in harmony with existing lighting conditions. In many situations, completely altering input lighting is undesirable for portrait retouching applications: one may want to preserve some authenticity in the captured environment. Existing shadow editing methods typically restrict their application to just the facial region and often offer limited lighting control options, such as shadow softening or rotation. In this paper, we introduce COMPOSE: a novel shadow editing pipeline for human portraits, offering precise control over shadow attributes such as shape, intensity, and position, all while preserving the original environmental illumination of the portrait. This level of disentanglement and controllability is obtained thanks to a novel decomposition of the environment map representation into ambient light and an editable gaussian dominant light source. COMPOSE is a four-stage pipeline that consists of light estimation and editing, light diffusion, shadow synthesis, and finally shadow editing. We define facial shadows as the result of a dominant light source, encoded using our novel gaussian environment map representation. Utilizing an OLAT dataset, we have trained models to: (1) predict this light source representation from images, and (2) generate realistic shadows using this representation. We also demonstrate comprehensive and intuitive shadow editing with our pipeline. Through extensive quantitative and qualitative evaluations, we have demonstrated the robust capability of our system in shadow editing.

Figures (13)

• Figure 1: Overview. COMPOSE is the first single-image portrait shadow editing method that achieves complete shadow editing control, i.e. adjusting shadow intensity, modifying light size, and changing shadow positions, all while preserving the source image's other lighting attributes (e.g. ambient light).
• Figure 2: Method Overview. COMPOSE is a $4$-stage shadow editing method consisting of single-image lighting estimation, light diffusion, shadow synthesis, and image compositing. By first estimating the dominant light position using the environment map regressed from the input image, COMPOSE can control shadow shape and intensity by controlling light spread and light intensity respectively as well as shadow position by changing the location of the dominant light. By estimating a diffuse image $\mathbf{I}_{D}$ as well as a shadowed image $\mathbf{I}_{S}$, COMPOSE can generate the final edited image $\mathbf{I}_{E}$ through image compositing.
• Figure 3: Shadow Synthesis Pipeline. Our shadow synthesis step consists of two stages: a U-Net followed by a conditional DDPM. The U-Net takes ambient image $\mathbf{I}_{D}$ and environment map $\mathbf{E}_{T}$ as input and outputs the coarse relit image $\mathbf{I}_{U}$. With $\mathbf{I}_{U}$ and $\mathbf{E}_{T}$ as spatial conditions, the conditional DDPM outputs the final shadowed image $\mathbf{I}_{S}$, refining shadow boundaries and improving image quality.
• Figure 4: Shadow Synthesis. Our model synthesizes more plausible shadows than the baselines, and, unlike face-relighting-with-geometrically-consistent-shadows, is also able to properly remove shadows from the source image before relighting. Environment maps are shown to help visualize each test lighting.
• Figure 5: Shadow Editing. Our method achieves complete shadow editing control, including softening/intensifying shadows, changing light size to alter shape and intensity, and changing light position, all while preserving the source image's ambient light.