PartCraft: Crafting Creative Objects by Parts

TL;DR

PartCraft tackles the challenge of fine-grained control in text-to-image generation by enabling users to select visual parts to craft novel objects. It combines unsupervised part discovery (via DINOv2-based clustering), a bottlenecked part-token representation, and an entropy-based attention loss to disentangle and precisely place parts during diffusion-based fine-tuning with LoRA adapters. The method demonstrates superior part reconstruction and composition on CUB-200-2011 and Stanford Dogs, with robust transferability to cross-domain prompts, and provides new metrics (EMR, CoSim) to quantify part disentanglement. This approach offers a practical path to creative generation by assembling parts from multiple concepts into coherent, plausible objects, with code and concepts released for broader adoption.

Abstract

This paper propels creative control in generative visual AI by allowing users to "select". Departing from traditional text or sketch-based methods, we for the first time allow users to choose visual concepts by parts for their creative endeavors. The outcome is fine-grained generation that precisely captures selected visual concepts, ensuring a holistically faithful and plausible result. To achieve this, we first parse objects into parts through unsupervised feature clustering. Then, we encode parts into text tokens and introduce an entropy-based normalized attention loss that operates on them. This loss design enables our model to learn generic prior topology knowledge about object's part composition, and further generalize to novel part compositions to ensure the generation looks holistically faithful. Lastly, we employ a bottleneck encoder to project the part tokens. This not only enhances fidelity but also accelerates learning, by leveraging shared knowledge and facilitating information exchange among instances. Visual results in the paper and supplementary material showcase the compelling power of PartCraft in crafting highly customized, innovative creations, exemplified by the "charming" and creative birds. Code is released at https://github.com/kamwoh/partcraft.

Figures (17)

• Figure 1: People often form creative concepts based on existing ones wilkenfeld2001similaritynagai2009conceptrunco2012standardbonnardel2005towards. For instance, a bird enthusiast may want to craft a unique bird with different parts (e.g., heads, bodies, and wings) from common bird types (e.g., blue jay, cardinal and song sparrow). (a) Using text prompts in T2I models often results in a lack of control and deviation from the intended details, especially those visual details that are difficult to describe. (b) While sketching is a direct way, not everyone possesses the ability to sketch, particularly in intricate detail. (c) Even with reference images, existing methods (e.g., DreamBooth ruiz2023dreambooth) did not consider learning object parts, thus cannot generate with desired parts. (d) Using additional control is even cumbersome, requiring too many inputs! (e) We aim to create an object by simply selecting desired parts. PartCraft learns from visual examples to generate the object with a faithful holistic structure, seamlessly integrating the chosen parts into a natural and coherent entity.
• Figure 2: Two sets of images were generated from their original parts (sources A and B). We can integrate a specific part (e.g., body, head, or even background (BG) of source B to target A seamlessly without effort.
• Figure 3: Overview of our PartCraft. All parts are organized into a dictionary, and their semantic embeddings are learned through a textual inversion approach. For instance, a text description like "a photo of a [Head,42] [Wing,87]..." guides the optimization of the corresponding textual embedding by reconstructing the associated image. To improve generation fidelity, we incorporate a bottleneck encoder $f$ (MLP) to compute the embedding $y$ (Eq. \ref{['eq:token_embs']}) as input to the text encoder. To promote disentanglement among learned parts, we minimize a specially designed attention loss, denoted as $\mathcal{L}_{attn}$.
• Figure 4: Part discovery within a semantic hierarchy involves partitioning each image into distinct parts and forming semantic clusters across unlabeled training data.
• Figure 5: Visual comparison under the part reconstruction setting. All images are generated by using the original parts of respective objects.