Revealing Unintentional Information Leakage in Low-Dimensional Facial Portrait Representations

TL;DR

This work analyzes unintentional information leakage from low-dimensional portrait attribute vectors produced by encoders under blackbox access, focusing on $f \in \mathbb{R}^{32}$ or $\mathbb{R}^{40}$. It introduces a reconstruction pipeline that couples a frozen StyleGAN generator with a trainable pre-generator $P$ to map $f$ into the generator’s input, optimizing a loss that combines pixel similarity and FaceNet embedding similarity, plus a distribution term to regularize the latent input. A key innovation is style mixing across StyleGAN layers, enabling ensemble outputs trained on different losses to be fused into distinct generator blocks for improved identity fidelity. Experiments on FFHQ and CelebA demonstrate that recognizable identity and other portrait attributes can be recovered (user studies show up to ~79–80% recognition), highlighting significant privacy risks in seemingly abstract feature representations and the need for cautious handling of such outputs.

Abstract

We evaluate the information that can unintentionally leak into the low dimensional output of a neural network, by reconstructing an input image from a 40- or 32-element feature vector that intends to only describe abstract attributes of a facial portrait. The reconstruction uses blackbox-access to the image encoder which generates the feature vector. Other than previous work, we leverage recent knowledge about image generation and facial similarity, implementing a method that outperforms the current state-of-the-art. Our strategy uses a pretrained StyleGAN and a new loss function that compares the perceptual similarity of portraits by mapping them into the latent space of a FaceNet embedding. Additionally, we present a new technique that fuses the output of an ensemble, to deliberately generate specific aspects of the recreated image.

Figures (5)

• Figure 1: The scenario of this paper: an encoder $E$ creates an attribute vector $f$ for an image, which is reconstructed by our decoder $D$.
• Figure 2: The in- and outputs of the three components of the reconstruction task: the encoder $E$ translates an image $X$ into a feature vector $f$, the pre-generator $P$ creates a generator input from the feature vector, and the generator $G$ turns the input into an image. The combination of $P$ and $G$ makes up the decoder. As described in Section \ref{['sec:pge']}, our training method is based on keeping $G$ frozen, but nonetheless comparing not $n$, but $X$ to a defined target.
• Figure 3: Stylemixing with a StyleGAN2, between an image $A$ (leftmost image) and an image $B$ (rightmost image). For each of the seven images between $A$ and $B$, all generator inputs are the same as for image $A$, with the exception of a single vector that was taken from image $B$. For the i-th of those seven images, we replaced the i-th input vector with the $B$ vector.
• Figure 4: Reconstructions for random images of the CelebA dataset, for a $D$ and $E$ both trained on FFHQ. From top to bottom, the rows present the target images, the result of the pixel-wise loss, the FaceNet embedding loss, and the fused version of the two. The final row displays the outcome of our predecessor, yang2019recAuxKnowledge. Details about training can be found in Section \ref{['sec:results_ffhq']}.
• Figure 5: Reconstructions for random images of the CelebA test set, for a $D$ trained on FFHQ and $E$ trained on the CelebA training set. From top to bottom, the rows present the target images, the result of the pixelwise loss, the FaceNet embedding loss, the fused version of the two and the results from yang2019recAuxKnowledge. Details about training can be found in Section \ref{['sec:results_celebA']}.