Any-Size-Diffusion: Toward Efficient Text-Driven Synthesis for Any-Size HD Images

TL;DR

This work tackles resolution-induced poor composition in text-to-image diffusion by introducing Any-Size-Diffusion (ASD), a two-stage pipeline that separates aspect-ratio adaptation (ARAD) from scalable upsampling (FSTD). ARAD trains on a curated set of aspect ratios to produce well-composed images across sizes, while FSTD upscales outputs to arbitrary resolutions using a novel implicit overlap tiled sampling that avoids seams and reduces inference time. The method achieves state-of-the-art quantitative metrics (lower FID, higher IS and CLIP scores) and enables 4K–8K outputs with substantially lower memory usage than baseline approaches. These results demonstrate practical viability for high-resolution, text-guided image synthesis with flexible sizing.

Abstract

Stable diffusion, a generative model used in text-to-image synthesis, frequently encounters resolution-induced composition problems when generating images of varying sizes. This issue primarily stems from the model being trained on pairs of single-scale images and their corresponding text descriptions. Moreover, direct training on images of unlimited sizes is unfeasible, as it would require an immense number of text-image pairs and entail substantial computational expenses. To overcome these challenges, we propose a two-stage pipeline named Any-Size-Diffusion (ASD), designed to efficiently generate well-composed images of any size, while minimizing the need for high-memory GPU resources. Specifically, the initial stage, dubbed Any Ratio Adaptability Diffusion (ARAD), leverages a selected set of images with a restricted range of ratios to optimize the text-conditional diffusion model, thereby improving its ability to adjust composition to accommodate diverse image sizes. To support the creation of images at any desired size, we further introduce a technique called Fast Seamless Tiled Diffusion (FSTD) at the subsequent stage. This method allows for the rapid enlargement of the ASD output to any high-resolution size, avoiding seaming artifacts or memory overloads. Experimental results on the LAION-COCO and MM-CelebA-HQ benchmarks demonstrate that ASD can produce well-structured images of arbitrary sizes, cutting down the inference time by 2x compared to the traditional tiled algorithm.

Figures (9)

• Figure 1: Resolution-induced Poor Composition. Given the text, (a) SD$_{2.1}$ and (b) MD$_{2.1}$, a MultiDiffusion model, raise poor composition issues in red boxes when synthesizing images of varying sizes, as opposed to (c) our ASD.
• Figure 2: The Any-Size-Diffusion (ASD) pipeline, including: 1) Stage-I, Any Ratio Adaptability Diffusion, translates text into images, adapting to various aspect ratios, and 2) is responsible for transforming low-resolution images from the Stage-I into high-resolution versions of any specified size. For procedure (c), the implicit overlap in tiled sampling, only the solid green line region is sent to the UNetModel for current denoising. At Stage-II, the dashed green arrow represents regions that are directly copied from the preceding denoised latents, potentially enhancing efficiency and consistency within the overall process.
• Figure 3: Comparison of various tiling strategies: (a) without overlapping, (b) with explicit overlapping, and (c) with implicit overlapping. Green tiles are explicit overlaps, and the orange tile is our implicit overlap at step t-1.
• Figure 4: Qualitative comparison of our proposed ASD method with other baselines, including (a) SR-Plus, (b) SR-Tile, (c) SR-Tile-Plus, (d) AR-Plus, (e) AR-Tile and (f) our proposed ASD. The yellow box indicates the resolution-induced poor composition. The orange box indicates better composition. The red solid line box is the zoom-in of the red dashed line box, aiming to inspect if there are any seaming issues. Our ASD outperforms others in both composition quality and inference time.
• Figure 5: Comparison of visual results. Composition quality of the text-to-image synthesis using (a) SD$_{2.1}$, a stable diffusion 2.1, (b) MD$_{2.1}$, a multi-diffusion based on SD 2.1, and (c) our ARAD. Color boxes indicate poor composition.