PokeFusion Attention: Enhancing Reference-Free Style-Conditioned Generation

TL;DR

This work targets reference-free style-conditioned character generation with diffusion models, where maintaining stable character geometry and consistent fine-grained style is challenging under text-only prompts. It introduces PokeFusion Attention, a lightweight decoder-level cross-attention mechanism that decouples text and learned style conditioning, training only the decoder attention and a compact style projection while keeping the backbone frozen. On a Pokemon-style dataset, it achieves higher style fidelity and semantic alignment than representative adapter-based baselines, without requiring reference images at inference and with a small parameter footprint. The method treats style as a distributional prior embedded in decoder attention, enabling portable, plug-and-play control across backbones and opening avenues for multi-style and broader-domain extensions.

Abstract

This paper studies reference-free style-conditioned character generation in text-to-image diffusion models, where high-quality synthesis requires both stable character structure and consistent, fine-grained style expression across diverse prompts. Existing approaches primarily rely on text-only prompting, which is often under-specified for visual style and tends to produce noticeable style drift and geometric inconsistency, or introduce reference-based adapters that depend on external images at inference time, increasing architectural complexity and limiting deployment flexibility.We propose PokeFusion Attention, a lightweight decoder-level cross-attention mechanism that fuses textual semantics with learned style embeddings directly inside the diffusion decoder. By decoupling text and style conditioning at the attention level, our method enables effective reference-free stylized generation while keeping the pretrained diffusion backbone fully frozen.PokeFusion Attention trains only decoder cross-attention layers together with a compact style projection module, resulting in a parameter-efficient and plug-and-play control component that can be easily integrated into existing diffusion pipelines and transferred across different backbones.Experiments on a stylized character generation benchmark (Pokemon-style) demonstrate that our method consistently improves style fidelity, semantic alignment, and character shape consistency compared with representative adapter-based baselines, while maintaining low parameter overhead and inference-time simplicity.

Figures (5)

• Figure 1: Task overview of reference-free style-conditioned character generation. Given only a text prompt, the goal is to generate stylized character images with consistent style, stable structure, and semantic alignment, without using reference images at inference time.
• Figure 2: The architecture of PokeFusion Attention. Only the red modules are trainable, while the main diffusion model remains frozen.
• Figure 3: Distribution of subject categories in the pokemon-blip-captions dataset based on caption keyword analysis.
• Figure 4: Qualitative comparison between IP-Adapter and PokeFusion Attention. Our method achieves more consistent shapes and stronger style fidelity without reference images.
• Figure 5: Robustness under varying inference settings. PokeFusion Attention shows more stable outputs than IP-Adapter.