Z-Magic: Zero-shot Multiple Attributes Guided Image Creator

TL;DR

This work tackles coherence in zero-shot multi-attribute image creation by reframing multi-attribute synthesis as conditional dependencies among attributes within score-based diffusion. It introduces Z-Magic, which sequentially conditions attributes to produce contextually aligned gradients $\nabla_{oldsymbol{x}_t}\log p(\boldsymbol{x}_t|\boldsymbol{\mathbf{c}}_1,...,\boldsymbol{\mathbf{c}}_n)$, and employs a Hessian-trick and multi-task learning (CAGrad) to manage complexity. The approach demonstrates superior coherence over independent-conditioning baselines across face and multi-modal generation tasks, with plug-and-play guidance using text, segmentation, landmarks, identity, content, and style. The method remains training-free, scalable to many attributes, and leverages energy-based guidance and pre-trained encoders to achieve robust, high-quality, attribute-coherent synthesis for AI-assisted design and creativity.

Abstract

The customization of multiple attributes has gained popularity with the rising demand for personalized content creation. Despite promising empirical results, the contextual coherence between different attributes has been largely overlooked. In this paper, we argue that subsequent attributes should follow the multivariable conditional distribution introduced by former attribute creation. In light of this, we reformulate multi-attribute creation from a conditional probability theory perspective and tackle the challenging zero-shot setting. By explicitly modeling the dependencies between attributes, we further enhance the coherence of generated images across diverse attribute combinations. Furthermore, we identify connections between multi-attribute customization and multi-task learning, effectively addressing the high computing cost encountered in multi-attribute synthesis. Extensive experiments demonstrate that Z-Magic outperforms existing models in zero-shot image generation, with broad implications for AI-driven design and creative applications.

Figures (9)

• Figure 1: We visualize the cosine similarity between gradients introduced by two conditions (i.e., landmark $g_0$ and face ID $g_1$) during the diffusion process of face synthesis. In the vanilla setting, each condition is applied separately without considering contextual coherence, causing the gradient directions to be nearly orthogonal, as random vectors in high-dimensional space are often orthogonal. In contrast, our method adjusts each condition based on the preceding ones, resulting in an obtuse-angled optimization direction where later conditions refine the gradients of earlier ones. In other words, our approach has two degrees of freedom to construct the gradient direction: angle and length.
• Figure 2: Illustration of the multi-condition optimization landscape with and without the our strategy (best viewed in color). Our approach $\nabla_{\mathbf{x}_t} \log p(\mathbf{c}_1,\mathbf{c}_2|\mathbf{x}_t)$ navigates the valleys, achieving lower condition losses, whereas the vanilla counterpart $\nabla_{\mathbf{x}_t} \log p(\mathbf{c}_1|\mathbf{x}_t)+\nabla_{\mathbf{x}_t} \log p(\mathbf{c}_2|\mathbf{x}_t)$ requires more steps to find a decreasing direction, resulting in higher loss values.
• Figure 3: We visualize the loss curves for generating a face image under three guided conditions using $\nabla_{\mathbf{x}_t} \log p(\mathbf{c}_1,\mathbf{c}_2,\mathbf{c}_3|\mathbf{x}_t)$, comparing results with vanilla $\sum_{i}\nabla_{\mathbf{x}_t} \log p(\mathbf{c}_i|\mathbf{x}_t)$. The proposed method achieves relatively lower metric values across all conditions, indicating a better balance in multi-condition control.
• Figure 4: Illustration of the effect of different condition sequences. When two conditions modify the same facial features, as shown in a), the sequence becomes important. However, if the conditions are weakly correlated, such as hair color and face parsing, the sequence has minimal impact as shown in b).
• Figure 5: Visualization of image creation guided by text and style reference images. All images are generated using the prompt “cat wearing glasses,” with style reference images displayed in the first column.