Zero-Shot Subject-Centric Generation for Creative Application Using Entropy Fusion

TL;DR

The paper tackles the challenge of generating high-quality subject-centric visuals without unwanted background elements in text-to-image pipelines. It introduces an entropy-weighted cross-attention fusion mechanism within a Flux MM-DiT diffusion framework and an LLM-driven agent system to automatically expand prompts and extract foreground keywords. By combining these components, the method yields accurate primary subject synthesis, precise alpha masks via GrabCut, and robust generation across diverse contexts, outperforming prior generation-then-matting and RGBA approaches. The approach is zero-shot, scalable to industrial applications, and supported by extensive qualitative, quantitative, and ablation evidence of its effectiveness and practicality.

Abstract

Generative models are widely used in visual content creation. However, current text-to-image models often face challenges in practical applications-such as textile pattern design and meme generation-due to the presence of unwanted elements that are difficult to separate with existing methods. Meanwhile, subject-reference generation has emerged as a key research trend, highlighting the need for techniques that can produce clean, high-quality subject images while effectively removing extraneous components. To address this challenge, we introduce a framework for reliable subject-centric image generation. In this work, we propose an entropy-based feature-weighted fusion method to merge the informative cross-attention features obtained from each sampling step of the pretrained text-to-image model FLUX, enabling a precise mask prediction and subject-centric generation. Additionally, we have developed an agent framework based on Large Language Models (LLMs) that translates users' casual inputs into more descriptive prompts, leading to highly detailed image generation. Simultaneously, the agents extract primary elements of prompts to guide the entropy-based feature fusion, ensuring focused primary element generation without extraneous components. Experimental results and user studies demonstrate our methods generates high-quality subject-centric images, outperform existing methods or other possible pipelines, highlighting the effectiveness of our approach.

Figures (8)

• Figure 1: Subject-centric images generated by our methods.
• Figure 2: The overview of our pipeline includes agents responsible for prompt extension and keyword extraction. The prompt serves as the input for the pretrained FLUX model, while the extracted keywords guide the alpha channel estimation using an entropy-based feature fusion strategy, ultimately facilitating the RGBA image generation process.
• Figure 3: Illustration of the attention map of the FLUX model for the generation process with the prompt "trick or treat" across different denoising steps. We use the Euler distance sampling method. The results show that at different timesteps, the noise contains varying amounts of information.
• Figure 4: Comparison with two pipelines: generation then segmentation/matting and RGBA image generation. Our method offers high-quality and precise generation of primary subject, outperforming other methods.
• Figure 5: Results generated from our generation framework with different LoRA styles.