Reflections on Diversity: A Real-time Virtual Mirror for Inclusive 3D Face Transformations

TL;DR

MOD tackles real-time, demographic-aware 3D face manipulation by integrating 3D Morphable Models with GAN-based texture editing in a seven-module pipeline. It jointly performs 3DMM fitting, geometry morphing, and texture transfer via FacialGAN, augmented by Collective Face averaging, to enable gender- and ethnicity-based transformations in an educational mirror. The authors provide qualitative demonstrations, quantitative comparisons against mobile filters and RT4Dface using FairFace classifications, and a user study, showing strengths in realism and demographic resemblance in several cases, while noting areas for improvement in underrepresented demographics. This work advances inclusive, interactive visual tools for diversity awareness and has practical implications for inclusive design in AR/VR and education.

Abstract

Real-time 3D face manipulation has significant applications in virtual reality, social media and human-computer interaction. This paper introduces a novel system, which we call Mirror of Diversity (MOD), that combines Generative Adversarial Networks (GANs) for texture manipulation and 3D Morphable Models (3DMMs) for facial geometry to achieve realistic face transformations that reflect various demographic characteristics, emphasizing the beauty of diversity and the universality of human features. As participants sit in front of a computer monitor with a camera positioned above, their facial characteristics are captured in real time and can further alter their digital face reconstruction with transformations reflecting different demographic characteristics, such as gender and ethnicity (e.g., a person from Africa, Asia, Europe). Another feature of our system, which we call Collective Face, generates an averaged face representation from multiple participants' facial data. A comprehensive evaluation protocol is implemented to assess the realism and demographic accuracy of the transformations. Qualitative feedback is gathered through participant questionnaires, which include comparisons of MOD transformations with similar filters on platforms like Snapchat and TikTok. Additionally, quantitative analysis is conducted using a pretrained Convolutional Neural Network that predicts gender and ethnicity, to validate the accuracy of demographic transformations.

Figures (10)

• Figure 1: Visualization of the linear shape identity model of the global LSFM model: (a) Mean shape ($\bar{\textbf{s}}$) and first 5 basis shapes (out of a total of $n_{id}$=158 basis shapes), each visualized as additions and subtractions away from the mean shape. In more detail, the top (bottom) row corresponds to setting the weight $p_i$ of the $i$-th basis shape to a positive (negative) value, corresponding to 3 standard deviations of its statistical distribution. (b) Generation of synthetic shapes through random sampling of the shape coefficients $\textbf{p}$, assuming a Gaussian distribution. Figure adapted from booth_large_2018.
• Figure 2: Pipeline of the proposed system.
• Figure 3: PCA projection of identity shape into the allowable range defined by box_scale_to_project
• Figure 4: (a) Adopted mark-up scheme of 68 facial landmarks (Figure from 10.1007/s11263-018-1134-y), (b) The process of mask generation. The first image shows the input face, the second illustrates the segmented face with different regions (mouth, nose, eyes, and eyebrows) covered by distinct masks and visualized by different colors.
• Figure 5: Installation setup with GUI, camera, lighting, and OpenGL rendering.