CLIP-FTI: Fine-Grained Face Template Inversion via CLIP-Driven Attribute Conditioning

TL;DR

This work addresses the privacy risks of leaked face templates by improving face template inversion (FTI) with CLIP-driven fine-grained attribute conditioning. CLIP-FTI derives region-specific facial attribute embeddings from CLIP and fuses them with the leaked template via a cross-modal attention module, mapping to StyleGAN3’s latent space to produce identity-consistent, attribute-faithful reconstructions. Empirical results show state-of-the-art identification accuracy and sharper component-level attributes across multiple backbones and datasets, with strong cross-model transferability. The study highlights the practical threat of backbone-agnostic inversions and underscores the need for defenses against CLIP-guided template inversion attacks.

Abstract

Face recognition systems store face templates for efficient matching. Once leaked, these templates pose a threat: inverting them can yield photorealistic surrogates that compromise privacy and enable impersonation. Although existing research has achieved relatively realistic face template inversion, the reconstructed facial images exhibit over-smoothed facial-part attributes (eyes, nose, mouth) and limited transferability. To address this problem, we present CLIP-FTI, a CLIP-driven fine-grained attribute conditioning framework for face template inversion. Our core idea is to use the CLIP model to obtain the semantic embeddings of facial features, in order to realize the reconstruction of specific facial feature attributes. Specifically, facial feature attribute embeddings extracted from CLIP are fused with the leaked template via a cross-modal feature interaction network and projected into the intermediate latent space of a pretrained StyleGAN. The StyleGAN generator then synthesizes face images with the same identity as the templates but with more fine-grained facial feature attributes. Experiments across multiple face recognition backbones and datasets show that our reconstructions (i) achieve higher identification accuracy and attribute similarity, (ii) recover sharper component-level attribute semantics, and (iii) improve cross-model attack transferability compared to prior reconstruction attacks. To the best of our knowledge, ours is the first method to use additional information besides the face template attack to realize face template inversion and obtains SOTA results.

Figures (9)

• Figure 1: Overview of CLIP-FTI. An original image $I$ yields a facial template $t$ (ArcFace/ElasticFace). A template$\to$semantic alignment adapter $TAA$ learns to predict $\hat{s}$ from $t$. A fusion mapping $M_{\text{FLP}}$ takes noise $n$, $t$, and $\hat{s}$ at attack time and outputs $\hat{w} \in \mathcal{W}$; the frozen StyleGAN3 synthesis network synthesizes a reconstructed image $\hat{I}=G(\hat{w})$.
• Figure 2: Training the template$\to$attribute alignment adapter $TAA$. The adapter learns to predict the aggregated semantic embedding $s$ from the facial template $t$ using MSE and cosine loss.
• Figure 3: Training of the fusion-latent projector $M_{FLP}$ and critic. A leaked template $t$, its TAA-predicted attribute embedding $\hat{s}$, and noise $n$ are fused by $M_{FLP}$ to produce a latent $\hat{w}$; the critic aligns $\hat{w}$ with the StyleGAN latent prior, while reconstruction losses between the generated image $\hat{I}=G(\hat{w})$ and the original image $I$ further supervise $M_{FLP}$.
• Figure 4: Internal architecture of $M_{\text{FLP}}$: three branches (noise $n$, template $t$, semantic embeddings $\hat{s}$) → attention fusion → MLP → latent $w \in W$.
• Figure 5: Qualitative comparison across datasets and reconstruction methods. The first row shows the original images, the second row shows reconstructions by Otroshi et al., and the third row presents the results of our CLIP-FTI. Column groups (left to right) correspond to two examples each from LFW, CelebA-HQ, and AgeDB.