Scale-DiT: Ultra-High-Resolution Image Generation with Hierarchical Local Attention

TL;DR

Scale-DiT tackles ultra-high-resolution text-to-image generation by addressing the dominant bottlenecks of quadratic attention and scarce high-resolution data. It introduces hierarchical local attention over fixed-size windows, guided by a low-resolution latent with scaled positional anchors to preserve global structure, and uses a light LoRA-based joint denoising scheme for efficient fusion of global and local cues. Token permutation via a Hilbert-curve and a fused-kernel tile-skipping mechanism yield GPU-friendly, near-linear scaling to $4K$ without additional high-resolution training data. Empirically, Scale-DiT delivers over $2\times$ faster inference and reduced memory usage compared with dense-attention baselines while achieving competitive or superior FID, IS, and CLIP metrics against state-of-the-art 4K methods. This work demonstrates a practical and generalizable path for scalable ultra-high-resolution diffusion-based generation.

Abstract

Ultra-high-resolution text-to-image generation demands both fine-grained texture synthesis and globally coherent structure, yet current diffusion models remain constrained to sub-$1K \times 1K$ resolutions due to the prohibitive quadratic complexity of attention and the scarcity of native $4K$ training data. We present \textbf{Scale-DiT}, a new diffusion framework that introduces hierarchical local attention with low-resolution global guidance, enabling efficient, scalable, and semantically coherent image synthesis at ultra-high resolutions. Specifically, high-resolution latents are divided into fixed-size local windows to reduce attention complexity from quadratic to near-linear, while a low-resolution latent equipped with scaled positional anchors injects global semantics. A lightweight LoRA adaptation bridges global and local pathways during denoising, ensuring consistency across structure and detail. To maximize inference efficiency, we repermute token sequence in Hilbert curve order and implement a fused-kernel for skipping masked operations, resulting in a GPU-friendly design. Extensive experiments demonstrate that Scale-DiT achieves more than $2\times$ faster inference and lower memory usage compared to dense attention baselines, while reliably scaling to $4K \times 4K$ resolution without requiring additional high-resolution training data. On both quantitative benchmarks (FID, IS, CLIP Score) and qualitative comparisons, Scale-DiT delivers superior global coherence and sharper local detail, matching or outperforming state-of-the-art methods that rely on native 4K training. Taken together, these results highlight hierarchical local attention with guided low-resolution anchors as a promising and effective approach for advancing ultra-high-resolution image generation.

Figures (12)

• Figure 1: Ultra-high-resolution images generated by Scale-DiT at $4K \times 4K$, $2K \times 3K$, and $2K \times 2K$. Zoomed-in regions highlight the fine-grained details preserved at these scales.
• Figure 2: Schematic of Scale-DiT's attention block modifications. The left column illustrates that high-resolution image latents $X$ are partitioned into local windows (red) in Hilbert-Curve order that attend to each other within their window. Simultaneously, a low-resolution guidance latent $X_{lr}$ (yellow) provides global context to each window via position scaling. The right column shows the joint-denoising process and the attention kernel, that $X, X_{lr}$ are processed together with LoRA applied on the $X_{lr}$ part. The attention mask (upper right) enforces local and guidance-specific interactions and tile skipping in attention calculation to enable efficient and coherent ultra-high-resolution generation. No additional high-resolution training data are needed.
• Figure 3: 4K comparison with leading baselines. Zoom in to observe the fine details. Scale-DiT produces high-resolution results with superior fidelity. Additional comparisons with other baselines are provided in the Appendix. The middle column shows methods finetuned on 4K data, while the right column shows training-free methods.
• Figure 4: Memory and speed statistics when scaling from 1K to 4K comparing with native FLUX.1 Dev with FlashAttention-2, Linear Attention used in SANA, and naive matmul attention. Our method's advantage in efficiency is more and more significant as resolution increases comparing to the others. OOM stands for out of memory.
• Figure 5: Ablation Study on attention scale. Images from left to right corresponds to cases that 1) High-res $X$ attends to both text and $X_{lr}$. 2) $X$ attends to only Text. 3) Each window of $X$ attends to its neighbors. 4) $X$ only attends to itself. 1) and 2) demonstrates the effectiveness of our low-res guidance, and 3) and 4) illustrates that with only allowing each window to attend to part of the neighbors will solve the boundary issues.