Durian: Dual Reference Image-Guided Portrait Animation with Attribute Transfer

Abstract

We present Durian, the first method for generating portrait animation videos with cross-identity attribute transfer from one or more reference images to a target portrait. Training such models typically requires attribute pairs of the same individual, which are rarely available at scale. To address this challenge, we propose a self-reconstruction formulation that leverages ordinary portrait videos to learn attribute transfer without explicit paired data. Two frames from the same video act as a pseudo pair: one serves as an attribute reference and the other as an identity reference. To enable this self-reconstruction training, we introduce a Dual ReferenceNet that processes the two references separately and then fuses their features via spatial attention within a diffusion model. To make sure each reference functions as a specialized stream for either identity or attribute information, we apply complementary masking to the reference images. Together, these two components guide the model to reconstruct the original video, naturally learning cross-identity attribute transfer. To bridge the gap between self-reconstruction training and cross-identity inference, we introduce a mask expansion strategy and augmentation schemes, enabling robust transfer of attributes with varying spatial extent and misalignment. Durian achieves state-of-the-art performance on portrait animation with attribute transfer. Moreover, its dual reference design uniquely supports multi-attribute composition and smooth attribute interpolation within a single generation pass, enabling highly flexible and controllable synthesis.

Figures (7)

• Figure 1: Portrait Animation with Attribute Transfer. Given a portrait image and single or multiple reference images specifying target attributes (e.g., hairstyle, eyeglasses), our method generates a portrait animation with facial attribute transfer conditioned on a keypoint sequence.
• Figure 2: Overview of Training Pipeline. Given an attribute-masked portrait image $\tilde{\mathbf{I}}_\mathrm{port}$ and an attribute-only image $\tilde{\mathbf{I}}_\mathrm{attr}$, Durian synthesizes a portrait animation with the transferred attribute. These inputs are constructed by randomly sampling two frames from a training video and applying the estimated masks. A sequence of facial keypoints $\{{\bm{k}_\tau}\}_{\tau = 1}^{F}$ is extracted from the video to guide the motion. During generation, spatial features from PRNet and ARNet are fused via spatial attention into the DNet, ensuring identity preservation and attribute consistency in the synthesized video.
• Figure 3: Aligned Attribute Mask Estimation. To improve attribute-portrait alignment, we estimate an aligned attribute mask via Face Aligner.
• Figure 4: Qualitative Comparison for Cross-Attribute Transfer. We compare our method and the baselines that combine X-Portrait xie2024x with StableHair zhang2025stablehair in cross-identity transfer setup. We provide more results in our Supp. Mat.
• Figure 5: Ablation Study. Omitting components or altering training scheme degrades visual quality.