HAIFAI: Human-AI Interaction for Mental Face Reconstruction

TL;DR

HAIFAI presents a two-stage interactive framework for reconstructing a face from a mental image by first inferring a StyleGAN2 latent code from explicit similarity rankings and then offering optional fine-grained edits via UP-FacE. The approach leverages a Siamese-transformer reconstruction network trained with a latent-embedding similarity objective and a computational user model to simulate human rankings, reducing data collection needs. Empirical evaluations show improvements over prior methods in embedding-consistent reconstruction, usability, and identification rate, including a new state-of-the-art 60.6% in lineup-based identification. The work advances human-AI interaction in cognitive tasks by balancing rapid, high-level AI exploration with precise human refinement, with potential forensic and interactive AI applications, while acknowledging biases in generative models and the need for real-world validation.

Abstract

We present HAIFAI - a novel two-stage system where humans and AI interact to tackle the challenging task of reconstructing a visual representation of a face that exists only in a person's mind. In the first stage, users iteratively rank images our reconstruction system presents based on their resemblance to a mental image. These rankings, in turn, allow the system to extract relevant image features, fuse them into a unified feature vector, and use a generative model to produce an initial reconstruction of the mental image. The second stage leverages an existing face editing method, allowing users to manually refine and further improve this reconstruction using an easy-to-use slider interface for face shape manipulation. To avoid the need for tedious human data collection for training the reconstruction system, we introduce a computational user model of human ranking behaviour. For this, we collected a small face ranking dataset through an online crowd-sourcing study containing data from 275 participants. We evaluate HAIFAI and an ablated version in a 12-participant user study and demonstrate that our approach outperforms the previous state of the art regarding reconstruction quality, usability, perceived workload, and reconstruction speed. We further validate the reconstructions in a subsequent face ranking study with 18 participants and show that HAIFAI achieves a new state-of-the-art identification rate of 60.6%. These findings represent a significant advancement towards developing new interactive intelligent systems capable of reliably and effortlessly reconstructing a user's mental image.

Figures (10)

• Figure 1: Our system HAIFAI first shows users sets of auxiliary faces over multiple rounds, asking them to rank these faces based on their resemblance to their mental image. A reconstruction network then predicts the latent vector corresponding to this mental image using the ranked auxiliary latents. A pre-trained generator decodes this vector to recreate the image. The target and reconstructed face images are then passed through a pre-trained embedding network to optimise our reconstruction network based on the similarity of their embeddings. The dotted red arrows indicate the gradient path to train our reconstruction network.
• Figure 2: Architecture of our reconstruction network. The reconstruction network processes a tuple of six auxiliary latent vectors ordered by user rankings for each iteration. For each tuple, a learnable 512-dimensional token (cls) is appended. These tuples are then fed into a Siamese transformer encoder, and the cls tokens are extracted at the end and passed through a linear layer. The resulting feature vectors are combined with another learnable token and processed by another transformer encoder. The class token is extracted again at the output and passed through a final linear layer to produce the reconstructed latent vector of the mental image.
• Figure 3: Three example images generated with a state-of-the-art StyleGAN2 karras2020analyzing generator. The mean absolute difference between the corresponding latent vectors $w_A$ and $w_B$ is higher compared to the difference between $w_B$ and $w_C$, although images A and B are visually more similar. However, when using face embeddings extracted with ArcFace deng2019arcface, the difference between the face embeddings $e_A$ and $e_B$ is smaller compared to $e_B$ and $e_C$.
• Figure 4: Agreement in face rankings among humans is shown in this matrix. Each cell $(i,j)$ represents the probability that two separate human raters will assign ranks $i$ and $j$ to the same face. A positive correlation is noted in human rankings, with an average Kendall's Tau value of 0.267. Additionally, humans tend to show higher consensus on the most and least similar faces, whereas the rankings in the middle range exhibit greater variability.
• Figure 5: Example reconstructions from our user study compared with the results from strohm2023usable. Each column shows the results for one participant. The first row shows the target faces participants had to memorise, and the following two rows show the reconstructions of the baselines. The last two columns show the initial reconstruction from HAIFAI after the first stage as well as the results from HAIFAI, where faces were further edited with UP-FacE strohm2024upface.