Controlling Face's Frame generation in StyleGAN's latent space operations: Modifying faces to deceive our memory
Agustín Roca, Nicolás Ignacio Britos
TL;DR
This work investigates how a face-frame, defined as the contour of the face including hair and ears but excluding facial features, is preserved or altered when editing faces in StyleGAN2 latent space. It introduces a face-frame variation metric based on image segmentation to quantify frame changes and presents a post-projection correction algorithm that perturbs latent codes with Gaussian noise to minimize frame deviation during image projection. By evaluating multiple latent directions (age, gender, orientation, eye/mouth openness, smile) and style mixing, the study finds that horizontal/vertical orientation, age, gender, and smile tend to distort the frame more, while eye-open and mouth-open edits preserve the frame more reliably. The findings have potential implications for memory research and eyewitness scenarios, suggesting which edits could be used to alter facial representations with minimal frame disruption, and point to future work in refining metrics, discovering new directions, and testing across other generative models. Overall, the approach provides a practical framework for maintaining face-frame fidelity while enabling attribute manipulation in face-generation tools.
Abstract
Innocence Project is a non-profitable organization that works in reducing wrongful convictions. In collaboration with Laboratorio de Sueño y Memoria from Instituto Tecnológico de Buenos Aires (ITBA), they are studying human memory in the context of face identification. They have a strong hypothesis stating that human memory heavily relies in face's frame to recognize faces. If this is proved, it could mean that face recognition in police lineups couldn't be trusted, as they may lead to wrongful convictions. This study uses experiments in order to try to prove this using faces with different properties, such as eyes size, but maintaining its frame as much as possible. In this project, we continue the work from a previous project that provided the basic tool to generate realistic faces using StyleGAN2. We take a deep dive into the internals of this tool to make full use of StyleGAN2 functionalities, while also adding more features, such as modifying certain of its attributes, including mouth-opening or eye-opening. As the usage of this tool heavily relies on maintaining the face-frame, we develop a way to identify the face-frame of each image and a function to compare it to the output of the neural network after applying some operations. We conclude that the face-frame is maintained when modifying eye-opening or mouth opening. When modifying vertical face orientation, gender, age and smile, have a considerable impact on its frame variation. And finally, the horizontal face orientation shows a major impact on the face-frame. This way, the Lab may apply some operations being confident that the face-frame won't significantly change, making them viable to be used to deceive subjects' memories.