Say Cheese! Detail-Preserving Portrait Collection Generation via Natural Language Edits

TL;DR

The paper tackles Portrait Collection Generation (PCG), enabling a reference portrait to be edited into a diverse, coherent collection through natural language edits. It introduces CHEESE, the first large-scale PCG dataset built with an LVLM-based annotation pipeline and inversion-based validation to ensure high-quality, compositional modification texts, and SCheese, a diffusion-based framework that balances complex, multi-attribute edits with strict identity and fine-grained detail preservation via Fusion IP-Adapter and ConsistencyNet with Decoupled-Attention. Extensive experiments on CHEESE show state-of-the-art performance in detail preservation and instruction following, with LVLM-based evaluations corroborating both qualitative and quantitative improvements over baselines. The work provides a solid foundation for personalized, privacy-conscious portrait editing and collection generation in real-world applications.

Abstract

As social media platforms proliferate, users increasingly demand intuitive ways to create diverse, high-quality portrait collections. In this work, we introduce Portrait Collection Generation (PCG), a novel task that generates coherent portrait collections by editing a reference portrait image through natural language instructions. This task poses two unique challenges to existing methods: (1) complex multi-attribute modifications such as pose, spatial layout, and camera viewpoint; and (2) high-fidelity detail preservation including identity, clothing, and accessories. To address these challenges, we propose CHEESE, the first large-scale PCG dataset containing 24K portrait collections and 573K samples with high-quality modification text annotations, constructed through an Large Vison-Language Model-based pipeline with inversion-based verification. We further propose SCheese, a framework that combines text-guided generation with hierarchical identity and detail preservation. SCheese employs adaptive feature fusion mechanism to maintain identity consistency, and ConsistencyNet to inject fine-grained features for detail consistency. Comprehensive experiments validate the effectiveness of CHEESE in advancing PCG, with SCheese achieving state-of-the-art performance.

Figures (14)

• Figure 1: Generated Portrait Collection Generation (PCG) Examples. Each group shows a reference image (left) and three generated results by our method with corresponding modification texts (below the photos). PCG introduces two key challenges: (1) Complex modifications requiring simultaneous changes in pose, camera viewpoint, and spatial layout; (2) Detail preservation maintaining fine-grained appearance characteristics, such as makeup, clothingand accessories.
• Figure 2: Generated Examples of Several Editing Challenges in PCG: camera distance, pose variation, and viewpoint transformation while preserving fine-grained details.
• Figure 3: The Dataset Construction Pipeline of CHEESE. (1) We first use the LVLM filters out near-duplicate pairs and pairs with excessive background changes. (2) For each filtered pair, an LVLM generates a natural-language modification text describing the transformation. (3) We then use the LVLM to inversion the target caption and compute CLIP score with the target image. We then iteratively refine the annotation if the score falls below a threshold. $IE$, $TE$ denote the image and text encoder of CLIP.
• Figure 4: Overview Architecture of SCheese:(Left) Our model consists of (1) DenoisingNet which is a main UNet that processes target image, (2) Fusion IP-Adapter that fuses high-level semantics of reference image $I_r$ and modification text $T_m$ , and (3) ConsistencyNet that encodes low-level features of $I_r$. (Right) We propose a Decoupled-Attention mechanism, which consists of a self-attention module and a cross-attention module. The outputs of these two modules are averaged together and further fused with features from the text encoder and Fusion IP-Adapter through a decoupled cross-attention layer.
• Figure 5: Qualitative Comparisons. Blue markers highlight successful detail preservation and instruction compliance; red boxes indicate the "copy-pasting" tendency; orange boxes denote blurriness or detail loss; purple boxes indicate limb abnormalities. SCheese successfully maintains reference details and image realism while executing complex modifications.