Controllable Shadow Generation with Single-Step Diffusion Models from Synthetic Data

TL;DR

This work tackles the challenge of generating realistic, controllable shadows for object-only images without a background, by leveraging a large synthetic dataset and a single-step diffusion model. The method conditions a diffusion denoiser on object geometry and light parameters $\mathcal{S}(\theta, \phi, s)$, using rectified flows to achieve fast, one-step shadow prediction. Key contributions include a scalable synthetic-data pipeline, a diffusion-based one-step shadow generator with explicit light-parameter conditioning, extensive ablations demonstrating the superiority of rectified-flow-based single-step inference, and a public shadow-generation benchmark for evaluation. The approach generalizes well to real images, enabling background-free shadow synthesis suitable for real-time compositing and visual effects.

Abstract

Realistic shadow generation is a critical component for high-quality image compositing and visual effects, yet existing methods suffer from certain limitations: Physics-based approaches require a 3D scene geometry, which is often unavailable, while learning-based techniques struggle with control and visual artifacts. We introduce a novel method for fast, controllable, and background-free shadow generation for 2D object images. We create a large synthetic dataset using a 3D rendering engine to train a diffusion model for controllable shadow generation, generating shadow maps for diverse light source parameters. Through extensive ablation studies, we find that rectified flow objective achieves high-quality results with just a single sampling step enabling real-time applications. Furthermore, our experiments demonstrate that the model generalizes well to real-world images. To facilitate further research in evaluating quality and controllability in shadow generation, we release a new public benchmark containing a diverse set of object images and shadow maps in various settings. The project page is available at https://gojasper.github.io/controllable-shadow-generation-project/

Figures (22)

• Figure 1: Our single-step model enables the generation of realistic shadows with precise control over their direction, softness, and intensity.
• Figure 2: Example renders with unprocessed (first two) and processed (last two) meshes. The red line represents the ground.
• Figure 3: The effect of the area light size on the shadow's softness. The area light sizes are 1, 2, and 3 respectively from left to right.
• Figure 4: Spherical coordinate system. $\theta$, $\phi$, and $r$ represent the polar angle, azimuthal angle, and the radius. $s$ corresponds to the size of the area light. We place the camera at negative $y$-axis.
• Figure 5: An example image from our dataset and its annotations.