IP-Prompter: Training-Free Theme-Specific Image Generation via Dynamic Visual Prompting

TL;DR

IP-Prompter introduces training-free visual prompting to enable theme-specific image generation (TSI) without fine-tuning. It combines Dynamic Visual Prompting (DVP) with an inpainting-based framework to extract intent, map textual elements to visual prompts via CLIP, and iteratively update a self-consistent prompt set, achieving high thematic fidelity and identity preservation. Quantitative and qualitative evaluations against state-of-the-art baselines demonstrate competitive performance in theme alignment and editable control, while enabling diverse applications such as multi-concept scenes, realistic characters in photos, and consistent storytelling. The approach offers a low-cost, flexible alternative for controllable image generation and highlights the potential of visual prompting to complement or replace training-heavy personalization methods.

Abstract

The stories and characters that captivate us as we grow up shape unique fantasy worlds, with images serving as the primary medium for visually experiencing these realms. Personalizing generative models through fine-tuning with theme-specific data has become a prevalent approach in text-to-image generation. However, unlike object customization, which focuses on learning specific objects, theme-specific generation encompasses diverse elements such as characters, scenes, and objects. Such diversity also introduces a key challenge: how to adaptively generate multi-character, multi-concept, and continuous theme-specific images (TSI). Moreover, fine-tuning approaches often come with significant computational overhead, time costs, and risks of overfitting. This paper explores a fundamental question: Can image generation models directly leverage images as contextual input, similarly to how large language models use text as context? To address this, we present IP-Prompter, a novel training-free TSI generation method. IP-Prompter introduces visual prompting, a mechanism that integrates reference images into generative models, allowing users to seamlessly specify the target theme without requiring additional training. To further enhance this process, we propose a Dynamic Visual Prompting (DVP) mechanism, which iteratively optimizes visual prompts to improve the accuracy and quality of generated images. Our approach enables diverse applications, including consistent story generation, character design, realistic character generation, and style-guided image generation. Comparative evaluations against state-of-the-art personalization methods demonstrate that IP-Prompter achieves significantly better results and excels in maintaining character identity preserving, style consistency and text alignment, offering a robust and flexible solution for theme-specific image generation.

Figures (11)

• Figure 1: Schematic illustration of visual promoting: (a) Text prompting in LLMs provides context and knowledge for the model to generate target content. (b) Existing personalized methods inject concepts into the model by fine-tuning the model, training a reference network, or altering the model structure to achieve thematic control. (c) Our proposed visual prompting based on inpainting represents a new model interaction paradigm, where visual prompts directly provides contextual information to the model, enabling fast and efficient controllable generation without the need to modify the generative model.
• Figure 2: Pipeline of IP-Prompter: Dynamic visual prompting (DVP) includes three key stages: (1) Comprehending user intent and extracting key elements; (2) Matching and generating visual prompts; and (3) Updating and evaluating prompts through self-consistency. This way (4) DVP enables effortless transition between diverse creative subjects, thereby enhancing the flexibility and efficiency of content generation.
• Figure 3: Attention maps computed during model inference. Here, a lots of visual prompts are combined in various arrangements. As shown in the upper right corner, areas with deeper colors are allocated more attention. These are referred to as significant regions and are marked with star symbols.
• Figure 4: Qualitative Results. We compare IP-Prompter with the SOTA personalization methods, including FLUX 1.0, Textual Inversion (TI) with SDXL, DreamBooth+LoRA with FLUX, Kolors Character with FLUX, IP-Adapter with FLUX, EasyRef, FreeCustom and ConsiStory.
• Figure 5: Quantitative evaluation and user study results.IP-Prompter achieves comparable scores to the fine-tuned FLUX model.