Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UniBioTransfer: A Unified Framework for Multiple Biometrics Transfer

Caiyi Sun, Yujing Sun, Xiangyu Li, Yuhang Zheng, Yiming Ren, Jiamin Wang, Yuexin Ma, Siu-Ming Yiu

Abstract

Deepface generation has traditionally followed a task-driven paradigm, where distinct tasks (e.g., face transfer and hair transfer) are addressed by task-specific models. Nevertheless, this single-task setting severely limits model generalization and scalability. A unified model capable of solving multiple deepface generation tasks in a single pass represents a promising and practical direction, yet remains challenging due to data scarcity and cross-task conflicts arising from heterogeneous attribute transformations. To this end, we propose UniBioTransfer, the first unified framework capable of handling both conventional deepface tasks (e.g., face transfer and face reenactment) and shape-varying transformations (e.g., hair transfer and head transfer). Besides, UniBioTransfer naturally generalizes to unseen tasks, like lip, eye, and glasses transfer, with minimal fine-tuning. Generally, UniBioTransfer addresses data insufficiency in multi-task generation through a unified data construction strategy, including a swapping-based corruption mechanism designed for spatially dynamic attributes like hair. It further mitigates cross-task interference via an innovative BioMoE, a mixture-of-experts based model coupled with a novel two-stage training strategy that effectively disentangles task-specific knowledge. Extensive experiments demonstrate the effectiveness, generalization, and scalability of UniBioTransfer, outperforming both existing unified models and task-specific methods across a wide range of deepface generation tasks. Project page is at https://scy639.github.io/UniBioTransfer.github.io/

UniBioTransfer: A Unified Framework for Multiple Biometrics Transfer

Abstract

Deepface generation has traditionally followed a task-driven paradigm, where distinct tasks (e.g., face transfer and hair transfer) are addressed by task-specific models. Nevertheless, this single-task setting severely limits model generalization and scalability. A unified model capable of solving multiple deepface generation tasks in a single pass represents a promising and practical direction, yet remains challenging due to data scarcity and cross-task conflicts arising from heterogeneous attribute transformations. To this end, we propose UniBioTransfer, the first unified framework capable of handling both conventional deepface tasks (e.g., face transfer and face reenactment) and shape-varying transformations (e.g., hair transfer and head transfer). Besides, UniBioTransfer naturally generalizes to unseen tasks, like lip, eye, and glasses transfer, with minimal fine-tuning. Generally, UniBioTransfer addresses data insufficiency in multi-task generation through a unified data construction strategy, including a swapping-based corruption mechanism designed for spatially dynamic attributes like hair. It further mitigates cross-task interference via an innovative BioMoE, a mixture-of-experts based model coupled with a novel two-stage training strategy that effectively disentangles task-specific knowledge. Extensive experiments demonstrate the effectiveness, generalization, and scalability of UniBioTransfer, outperforming both existing unified models and task-specific methods across a wide range of deepface generation tasks. Project page is at https://scy639.github.io/UniBioTransfer.github.io/
Paper Structure (43 sections, 6 equations, 18 figures, 11 tables, 2 algorithms)

This paper contains 43 sections, 6 equations, 18 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. DeepFace Generation
  4. Reference-Based Image Editing
  5. Mixture-of-Experts (MoE) and Multi-task Learning
  6. Method
  7. A Unified Data Construction Strategy
  8. Relative-Static Attributes Corruption
  9. Spatially-Dynamic Attributes Corruption
  10. Remarks.
  11. BioMoE
  12. Structure-Aware Routing
  13. Two-Stage Training Strategy
  14. Stage I: Task-Specific Pre-training
  15. Stage II: Task-Unified Fine-tuning
  16. ...and 28 more sections

Figures (18)

  • Figure 2: Limitations of traditional mask-based strategy for attributes with significant structural changes (e.g., hair transfer). Masking exposes ground-truth geometry (a-top), causing models to learn only inpainting rather than true shape transfer. Our swapping-based strategy removes silhouette information in the target (b-top), forcing the network to transfer shape from the reference.
  • Figure 3: Our unified data corruption strategy for different attribute types. (a) Relative-static attributes: the target is constructed by simple masking or data augmentation of the GT image. (b) Spatially-dynamic attributes: we utilize our swapping-based corruption strategy, which employs an off-the-shelf generative model to replace specific attributes in the GT with arbitrary novel variations, preventing shape leakage from mask boundaries.
  • Figure 4: UniBioTransfer architecture overview. (a) Overall framework. (b) We introduce an MoE-enhanced Feed Forward Network (FFN). (c) Expert selection is guided by a Structure-Aware Router. (d) The entire system is optimized using a two-stage training strategy designed to stabilize routing and promote expert specialization.
  • Figure 4: Ablation studies on model design and training strategy. For each task, we report its most representative metric (identity similarity, hair clip distance, pose distance, and head clip distance respectively).
  • Figure 5: Visual Comparisons on diverse deepface tasks. More results in Suppl.
  • ...and 13 more figures