SPF-Portrait: Towards Pure Text-to-Portrait Customization with Semantic Pollution-Free Fine-Tuning

TL;DR

SPF-Portrait tackles semantic pollution during fine-tuning of text-to-portrait diffusion models by introducing a two-stage, dual-path framework that preserves the original model's behavior while actively learning target semantics. A Reference Path (frozen original model) and a Response Path (fine-tuned model) are trained with a Semantic-Aware Fine Control Map (SFCM) to localize and constrain alignment, plus a difference-vector–based response enhancement to bridge cross-modal gaps. The approach combines diffusion-model losses, cross-attention and UNet feature alignment, and CLIP-guided region weighting, yielding state-of-the-art preservation and responsiveness across extensive qualitative and quantitative evaluations on a large portrait dataset. The method demonstrates incremental learning and robustness to continuous text edits, with promising extensions to broader text-to-image tasks and possible text-to-video applications.

Abstract

Fine-tuning a pre-trained Text-to-Image (T2I) model on a tailored portrait dataset is the mainstream method for text-to-portrait customization. However, existing methods often severely impact the original model's behavior (e.g., changes in ID, layout, etc.) while customizing portrait attributes. To address this issue, we propose SPF-Portrait, a pioneering work to purely understand customized target semantics and minimize disruption to the original model. In our SPF-Portrait, we design a dual-path contrastive learning pipeline, which introduces the original model as a behavioral alignment reference for the conventional fine-tuning path. During the contrastive learning, we propose a novel Semantic-Aware Fine Control Map that indicates the intensity of response regions of the target semantics, to spatially guide the alignment process between the contrastive paths. It adaptively balances the behavioral alignment across different regions and the responsiveness of the target semantics. Furthermore, we propose a novel response enhancement mechanism to reinforce the presentation of target semantics, while mitigating representation discrepancy inherent in direct cross-modal supervision. Through the above strategies, we achieve incremental learning of customized target semantics for pure text-to-portrait customization. Extensive experiments show that SPF-Portrait achieves state-of-the-art performance. Project page: https://spf-portrait.github.io/SPF-Portrait/

Figures (16)

• Figure 1: Visualization of the Attention Map. The salient regions directly reflect response intensity to the target semantics "a hat".
• Figure 2: The Dual-Path Contrastive Learning Pipeline of SPF-Portrait. The text in blue is the Base text, while those in red is the Target text. Reference Path takes only Base text as input, while Response Path takes complete text (Base text & Target text) as input.
• Figure 3: Analysis of Alignment Process. (a) Vanilla alignment results in the over-alignment with original portrait. (b) For the same customization attribute, reference image-based fine-tuning offers a more distinct target response region than T2I fine-tuning.
• Figure 4: Comparison with Traditional Supervision on Image Fidelity.(a) illustrates the trend of Image-Reward (IR) and CLIP Score (CLIP-T) across training steps. Image-Reward xu2023imagereward is a metric used to evaluate image fidelity. (b) displays samples from traditional method avrahami2022blended and ours.
• Figure 5: Qualitative Comparisons with SOTA methods. We compare ours with naive fine-tuning rombach2022high, PEFT-based methods (LoRA hu2021lora, AdaLoRA zhang2023adalora ) and the decoupled methods (Tokencompose wang2024tokencompose, Magenet zhuang2024magnet).