Deformable One-shot Face Stylization via DINO Semantic Guidance

TL;DR

This work tackles one-shot deformable face stylization by leveraging a real-style paired reference and a DINO-based semantic guidance to learn cross-domain structural deformation. A deformation-aware StyleGAN2 with TPS-STN modules is fine-tuned using two novel cross-domain losses—directional deformation and relative structural consistency—alongside an adversarial style transfer component and color alignment via style-mixing. The framework achieves expressive geometric exaggerations while preserving identity, with efficient training (~10 minutes) and strong qualitative, quantitative, and user-study results surpassing state-of-the-art one-shot methods. The approach demonstrates practical potential for flexible, high-fidelity stylization in real-world applications where paired references are available.

Abstract

This paper addresses the complex issue of one-shot face stylization, focusing on the simultaneous consideration of appearance and structure, where previous methods have fallen short. We explore deformation-aware face stylization that diverges from traditional single-image style reference, opting for a real-style image pair instead. The cornerstone of our method is the utilization of a self-supervised vision transformer, specifically DINO-ViT, to establish a robust and consistent facial structure representation across both real and style domains. Our stylization process begins by adapting the StyleGAN generator to be deformation-aware through the integration of spatial transformers (STN). We then introduce two innovative constraints for generator fine-tuning under the guidance of DINO semantics: i) a directional deformation loss that regulates directional vectors in DINO space, and ii) a relative structural consistency constraint based on DINO token self-similarities, ensuring diverse generation. Additionally, style-mixing is employed to align the color generation with the reference, minimizing inconsistent correspondences. This framework delivers enhanced deformability for general one-shot face stylization, achieving notable efficiency with a fine-tuning duration of approximately 10 minutes. Extensive qualitative and quantitative comparisons demonstrate our superiority over state-of-the-art one-shot face stylization methods. Code is available at https://github.com/zichongc/DoesFS

Figures (12)

• Figure 1: We propose a deformation-aware face stylization framework trained on a single real-style image pair (left). Our framework can generate diverse, high-quality, stylized faces with desired deformations, while maintaining the input identity well (right).
• Figure 2: GAN inversion. Previous one-shot stylization methods like zhu2022mtg and chong2022jojogan, build the cross-domain guidance by inverting the artistic style image into real face domain. But compared with the ground truth real face, current GAN inversion techniques tov2021designingabdal2019image2stylegan still cannot give out a faithful mapping across unseen domains, which may mislead the structure deformation in the stylization.
• Figure 3: Visualization of the hierarchical features from CLIP radford2021clip, FaRL zheng2022farl, and DINO caron2021emerging, where the same color represents the same semantics shared. Note that the same architecture ViT (ViT-B/16) is employed fairly for image encoding. We choose layers 3, 6, and 12 to represent different levels (L, M, H) of features. Following amir2021deep, we only use keys and tokens in ViTs, while discarding the queries and values. The [CLS] token is also discarded as it mainly encodes the visual appearance tumanyan2022splicing.
• Figure 4: Framework Overview. Given a single real-style paired reference, we fine-tune a deformation-aware generator $G^t$ that simultaneously realizes geometry deformation and appearance transfer. To learn the cross-domain deformation, we design a directional deformation loss $\mathcal{L}_{direct}$ and a relative structural consistency loss $\mathcal{L}_{cons}$, both computed in DINO feature space (middle). Inversion and style mixing further ensure a consistent DINO semantic representation aligned with the given reference (left). In addition, we involve adversarial training using a patch-level discriminator to enhance the transferred style and fidelity (right).
• Figure 5: Deformation-aware generator. (a) A StyleGAN synthesis block plugged in with a Transform module. (b) A thin-plate-spline spatial transformer (TPS-STN) for feature warping, which shows the detail of the Transform module in (a).