Illumination and Shadows in Head Rotation: experiments with Denoising Diffusion Models

TL;DR

This work investigates continuous head-rotation editing in the latent space of pre-trained denoising diffusion models, emphasizing lighting and shadow effects. By labeling CelebA images by illumination direction and using DDIM embeddings, the authors compute two linear trajectories in latent space (left and right) via centroid regression within attribute-homogeneous subsets, achieving rotations up to ±$30^{\circ}$ without retraining the model. The approach yields qualitative demonstrations of smooth, illumination-aware edits and highlights the critical role of lighting in preserving identity during rotation, while also acknowledging notable limitations such as identity preservation, artifacts, and contour deformations. The authors release the method and illumination labels as open-source resources, paving the way for further quantitative evaluation, video-based extensions, and broader holistic image manipulations in generative models.

Abstract

Accurately modeling the effects of illumination and shadows during head rotation is critical in computer vision for enhancing image realism and reducing artifacts. This study delves into the latent space of denoising diffusion models to identify compelling trajectories that can express continuous head rotation under varying lighting conditions. A key contribution of our work is the generation of additional labels from the CelebA dataset,categorizing images into three groups based on prevalent illumination direction: left, center, and right. These labels play a crucial role in our approach, enabling more precise manipulations and improved handling of lighting variations. Leveraging a recent embedding technique for Denoising Diffusion Implicit Models (DDIM), our method achieves noteworthy manipulations, encompassing a wide rotation angle of $\pm 30$ degrees, while preserving individual distinct characteristics even under challenging illumination conditions. Our methodology involves computing trajectories that approximate clouds of latent representations of dataset samples with different yaw rotations through linear regression. Specific trajectories are obtained by analyzing subsets of data that share significant attributes with the source image, including light direction. Notably, our approach does not require any specific training of the generative model for the task of rotation; we merely compute and follow specific trajectories in the latent space of a pre-trained face generation model. This article showcases the potential of our approach and its current limitations through a qualitative discussion of notable examples. This study contributes to the ongoing advancements in representation learning and the semantic investigation of the latent space of generative models.

Figures (24)

• Figure S1: Rotation examples. The sources are images 114 16399, and 98018 of CelebA (central image).
• Figure S2: The U-net architecture of our denoising model.
• Figure S3: Main architectural modules, including the residual block, down block and up block.
• Figure S4: Distribution of CelebA attributes. In CelebA, each attribute is annotated with either -1 or 1. For example, for gender, "male = 1" stands for male, and "male = -1" stands for female.
• Figure S5: (a) Yaw, Pitch and Roll angles in HPE. (b) Examples of head pose estimation for CelebA images: yaw is in green, pitch in blue, and roll in red.