Layered Diffusion Model for One-Shot High Resolution Text-to-Image Synthesis

TL;DR

The paper tackles high-resolution text-to-image diffusion synthesis in a one-shot framework, addressing inefficiencies of cascaded or solely high-resolution models. It introduces a Layered Diffusion Model built on a layered U-Net that outputs at multiple resolutions within a single forward pass, leveraging $\mathcal{S}$ for noise scaling via sinc interpolation and $\log$-scaled shifted cosine schedules. Key contributions include improved Fréchet Inception Distance and Inception Score at $256\times256$ and $512\times512$ compared to single-resolution baselines, reduced per-step computation (FLOPs), and training optimizations such as strategic cropping and model stacking. This approach eliminates the need for separate super-resolution modules, offering a more efficient pathway to high-quality, diverse, text-conditioned high-resolution imagery.

Abstract

We present a one-shot text-to-image diffusion model that can generate high-resolution images from natural language descriptions. Our model employs a layered U-Net architecture that simultaneously synthesizes images at multiple resolution scales. We show that this method outperforms the baseline of synthesizing images only at the target resolution, while reducing the computational cost per step. We demonstrate that higher resolution synthesis can be achieved by layering convolutions at additional resolution scales, in contrast to other methods which require additional models for super-resolution synthesis.

Figures (1)

• Figure 1: Layered diffusion model architecture. We provide noisy inputs at 3 resolutions and generate images of corresponding size. Note that information is only shared between layers during upsampling via skip connection.