KPLM-STA: Physically-Accurate Shadow Synthesis for Human Relighting via Keypoint-Based Light Modeling

TL;DR

This work tackles the problem of generating realistic and geometrically coherent shadows for composite images of humans during relighting. It introduces KPLM-STA, combining a Keypoints Linear Model (KPLM) that uses nine keypoints plus a trunk block with a Shadow Triangle Algorithm (STA) to capture limb-level shadow geometry, which then informs a diffusion-based shadow generator conditioned by geometric priors via ControlNet, followed by GAN-based post-processing. The approach achieves state-of-the-art results on DESOBA, DESOBAv2, and demonstrates generalization to IC-Light relighting, improving both appearance realism and geometric accuracy of shadows. Ablation studies confirm the contributions of KPLM and STA, and results indicate strong practical impact for photo-realistic compositing and multi-directional relighting scenarios.

Abstract

Image composition aims to seamlessly integrate a foreground object into a background, where generating realistic and geometrically accurate shadows remains a persistent challenge. While recent diffusion-based methods have outperformed GAN-based approaches, existing techniques, such as the diffusion-based relighting framework IC-Light, still fall short in producing shadows with both high appearance realism and geometric precision, especially in composite images. To address these limitations, we propose a novel shadow generation framework based on a Keypoints Linear Model (KPLM) and a Shadow Triangle Algorithm (STA). KPLM models articulated human bodies using nine keypoints and one bounding block, enabling physically plausible shadow projection and dynamic shading across joints, thereby enhancing visual realism. STA further improves geometric accuracy by computing shadow angles, lengths, and spatial positions through explicit geometric formulations. Extensive experiments demonstrate that our method achieves state-of-the-art performance on shadow realism benchmarks, particularly under complex human poses, and generalizes effectively to multi-directional relighting scenarios such as those supported by IC-Light.

Figures (9)

• Figure 1: Comparison with GPSDiffusion gpsdiff and IC-Light iclight. The first row and columns $2-4$ are the random different results of GPSDiffusion, the second row and columns $2-3$ are the different light results of IC-Light. It can be seen that each image presents problems of appearance realism (shadow color is slightly darker) and geometric accuracy (limb shape is incorrect). However, our method (ours) has effectively solved these two problems.
• Figure 2: The framework of our KPLM-STA. In the first stage, we use KPLM to get key point coordinates, use STA to obtain the shadow angle. In the second stage, we use Control Encoder and Diffusion to generate image. Finally, we use a post-processing GAN for realistic processing.
• Figure 3: KPLM modeling schematic diagram.
• Figure 4: KPLM compared with the mask of object in the most universal method OpenPose, MediaPipe.
• Figure 5: Shadow Triangle Construction. Given a human limb segment AB and the corresponding shadow endpoint $C$, the triangle $\triangle ABC$ is formed under parallel light from the sun. $A'$ and $B'$ denote projected shadow points on the ground. The angle $\theta$ is formed between the light direction and the ground plane.