HiPrompt: Tuning-free Higher-Resolution Generation with Hierarchical MLLM Prompts

TL;DR

HiPrompt tackles the problem of generating ultra-high-resolution images with diffusion models without fine-tuning by introducing hierarchical prompts that couple global user descriptions with patch-level MLLM-derived prompts. It couples this with a noise decomposition mechanism that separates the diffusion process into low- and high-frequency components, enabling parallel denoising guided by global and local semantics. The approach reduces object repetition and enhances structural and textural fidelity across resolutions up to $4096\times 4096$, outperforming state-of-the-art tuning-free methods on both quantitative metrics (FID$_r$, FID$_c$, KID$_r$, KID$_c$, IS$_r$, CLIP) and qualitative visual assessments. The method is versatile, leveraging multiple MLLMs (e.g., LLAVA, ShareCaptioner) and promising broad applicability for high-fidelity, high-resolution image synthesis without model fine-tuning.

Abstract

The potential for higher-resolution image generation using pretrained diffusion models is immense, yet these models often struggle with issues of object repetition and structural artifacts especially when scaling to 4K resolution and higher. We figure out that the problem is caused by that, a single prompt for the generation of multiple scales provides insufficient efficacy. In response, we propose HiPrompt, a new tuning-free solution that tackles the above problems by introducing hierarchical prompts. The hierarchical prompts offer both global and local guidance. Specifically, the global guidance comes from the user input that describes the overall content, while the local guidance utilizes patch-wise descriptions from MLLMs to elaborately guide the regional structure and texture generation. Furthermore, during the inverse denoising process, the generated noise is decomposed into low- and high-frequency spatial components. These components are conditioned on multiple prompt levels, including detailed patch-wise descriptions and broader image-level prompts, facilitating prompt-guided denoising under hierarchical semantic guidance. It further allows the generation to focus more on local spatial regions and ensures the generated images maintain coherent local and global semantics, structures, and textures with high definition. Extensive experiments demonstrate that HiPrompt outperforms state-of-the-art works in higher-resolution image generation, significantly reducing object repetition and enhancing structural quality.

Figures (6)

• Figure 1: Examples of HiPrompt at various higher resolutions based upon SDXL. SDXL can synthesize images up to a resolution of $1024^2$, while our method extends SDXL to generate images at $4\times$, $16\times$ without any fine-tuning. Please zoom in for a better view.
• Figure 2: Visual comparison between ScaleCrafter he2023scalecrafter, DemoFusion du2024demofusion, AccDiffusion lin2024accdiffusion, and HiPrompt. Under setting of $16\times$ ($4096^2$). The red boxes highlight the repeated object problem, while the yellow boxes denote areas with blurred and unreasonable structures.
• Figure 3: The overall framework of HiPrompt. The upper portion illustrates the hierarchical prompt generation module, while the lower section outlines the noise decomposition process. Given a low-resolution image, MLLMs are employed to generate dense local descriptions for each overlapping local patch. To enhance the quality of these detailed prompts, we utilize N-grams $(n = 1)$ refinement to filter out irrelevant noise. Subsequently, HiPrompt decomposes the noisy image into low- and high-spatial frequency components using low-pass and high-pass Gaussian filters. These components are denoised in parallel, conditioned on the hierarchical prompts, and then summarized into final estimation during the inverse denoising process.
• Figure 4: Comparion of hierarchical prompts from LLAVA liu2024improved and ShareCaptioner chen2023sharegpt4v. MLLM generates a dense local prompt $\mathbf{y}_{i}^j$ to describe details and textures of each local patch $\mathbf{p}_{i}^j$.
• Figure 5: Qualitative comparison with other baselines. The red boxes highlight the repeated small objects, and the yellow boxes denote blurred areas and unreasonable structures.