PSTF-AttControl: Per-Subject-Tuning-Free Personalized Image Generation with Controllable Face Attributes

TL;DR

This work tackles the challenge of achieving precise, per-subject-tuning-free control over facial attributes while preserving identity in personalized image generation. It combines a face-recognition identity extractor with a StyleGAN2 $W^+$ latent-space mapping via the $e4e$ encoder and introduces a Triplet-Decoupled Cross-Attention module to fuse identity, attribute, and text embeddings within a diffusion UNet. The method uses attribute-controlled data augmentation with edit directions in $W^+$ and a ControlNet conditioned on identity features, trained on FFHQ without per-identity fine-tuning. Empirical results show improved identity preservation and finer-grained attribute control compared with tuning-based methods and other PSTF approaches, with code released publicly for replication.

Abstract

Recent advancements in personalized image generation have significantly improved facial identity preservation, particularly in fields such as entertainment and social media. However, existing methods still struggle to achieve precise control over facial attributes in a per-subject-tuning-free (PSTF) way. Tuning-based techniques like PreciseControl have shown promise by providing fine-grained control over facial features, but they often require extensive technical expertise and additional training data, limiting their accessibility. In contrast, PSTF approaches simplify the process by enabling image generation from a single facial input, but they lack precise control over facial attributes. In this paper, we introduce a novel, PSTF method that enables both precise control over facial attributes and high-fidelity preservation of facial identity. Our approach utilizes a face recognition model to extract facial identity features, which are then mapped into the $W^+$ latent space of StyleGAN2 using the e4e encoder. We further enhance the model with a Triplet-Decoupled Cross-Attention module, which integrates facial identity, attribute features, and text embeddings into the UNet architecture, ensuring clean separation of identity and attribute information. Trained on the FFHQ dataset, our method allows for the generation of personalized images with fine-grained control over facial attributes, while without requiring additional fine-tuning or training data for individual identities. We demonstrate that our approach successfully balances personalization with precise facial attribute control, offering a more efficient and user-friendly solution for high-quality, adaptable facial image synthesis. The code is publicly available at https://github.com/UnicomAI/PSTF-AttControl.

Figures (8)

• Figure 1: Overview of PSTF-AttControl framework. We use the StyleGAN2 encoder to extract facial attribute features and integrate face, attribute, and text embeddings into the diffusion model via the Triplet-Decoupled Cross-Attention module. The attribute-controlled synthesis approach for data augmentation enables the model to learn controllable facial attribute editing. We train only the projection and the Cross-Attention of facial attributes, shown in the pink modules in the figure.
• Figure 2: The Results of PSTF-AttControl. By modifying the facial attribute components in the $W^+$ space, PSTF-AttControl enables the continuous generation of personalized images with varying attribute strengths. Here, we showcase the generation results for 12 different attributes using the faces of Fei-Fei Li and Yann LeCun as examples, demonstrating the effectiveness of our method in producing diverse, high-quality variations based on facial attributes.
• Figure 3: Comparison with PreciseControl. For the "smile" attribute, our method produces higher-quality teeth generation compared to PreciseControl (highlighted in blue). In the case of the "eyeclose" attribute, PreciseControl fails to make any visible changes, whereas our method smoothly closes the eyes (highlighted in red). The final column shows the mask used by PreciseControl.
• Figure 4: Comparison with PSTF Methods. The results from PuLID and InstantID show that text-based control of facial attributes is limited in its effectiveness. The results produced by W+Adapter show limited similarity to the reference face, and its manipulation of the “beard” attribute lacks precision. In contrast, our method successfully generates the desired facial features while maintaining consistent identity across faces.
• Figure 5: Some results of comparison with InstantID, W+Adapter and PuLID on the Unsplash-50 dataset. Our method outperformed others in preserving facial identity and fine details.