Zero-Shot Visual Concept Blending Without Text Guidance

TL;DR

The paper tackles controlling which features from reference images are transferred to a source image in zero-shot image generation. It introduces Visual Concept Blending, a four-step pipeline that operates in a partially disentangled CLIP embedding space via IP-Adapter to extract and merge common or distinct features from two references without fine-tuning or text prompts. Key contributions include a concrete embedding-based framework, demonstrations across style transfer, metamorphosis, and abstract concept transformations, and a user study validating feature transfer interpretability. The results show robust, text-free control over high-level attributes and highlight practical potential for art and design, while acknowledging limitations in embedding disentanglement and parameter sensitivity. Future work aims at improving disentanglement, automatic parameter tuning, and broader domain applicability.

Abstract

We propose a novel, zero-shot image generation technique called "Visual Concept Blending" that provides fine-grained control over which features from multiple reference images are transferred to a source image. If only a single reference image is available, it is difficult to isolate which specific elements should be transferred. However, using multiple reference images, the proposed approach distinguishes between common and unique features by selectively incorporating them into a generated output. By operating within a partially disentangled Contrastive Language-Image Pre-training (CLIP) embedding space (from IP-Adapter), our method enables the flexible transfer of texture, shape, motion, style, and more abstract conceptual transformations without requiring additional training or text prompts. We demonstrate its effectiveness across a diverse range of tasks, including style transfer, form metamorphosis, and conceptual transformations, showing how subtle or abstract attributes (e.g., brushstroke style, aerodynamic lines, and dynamism) can be seamlessly combined into a new image. In a user study, participants accurately recognized which features were intended to be transferred. Its simplicity, flexibility, and high-level control make Visual Concept Blending valuable for creative fields such as art, design, and content creation, where combining specific visual qualities from multiple inspirations is crucial.

Figures (14)

• Figure 1: Conventional visual concept translation vs. the proposed visual concept blending. In conventional methods, it is difficult to distinguish the features that should be transferred because only a single reference image is available. Our proposed method can effectively control the features we wish to transfer (e.g., illustration style or tiger stripes) by performing operations in the projection space of the CLIP embedding vectors and extracting a common concept from multiple reference images.
• Figure 2: Architecture of the proposed method.
• Figure 3: Parameter sensitivity analysis. Here, $\theta$ is the threshold for extracting reference features, and $d$ is the strength of the depth constraint.
• Figure 4: Blending the common features of reference images and comparison with existing single-reference methods. For VCB, $\theta=0.4$ and $d=0.0, 0.0, 0.1, 0.6$, respectively.
• Figure 5: Blending the distinct features between reference images.$\theta=0.6$ and $d=0.0, 0.0, 1.0$, respectively.