STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesis

TL;DR

STARFlow demonstrates that normalizing-flow models can scale to high-resolution image synthesis by combining Transformer Autoregressive Flows with a deep–shallow architecture and latent-space learning. It establishes universality for multi-block AFs and introduces a latent prior with a trainable pixel decoder, enabling end-to-end maximum-likelihood training in continuous space. A principled classifier-free guidance formulation and a training-free inpainting/interactive editing pipeline further enhance sample quality and utility. When evaluated against diffusion and AR baselines, STARFlow achieves competitive FID scores and robust text-conditioned generation while offering advantages in likelihood evaluation and inference throughput under practical constraints.

Abstract

We present STARFlow, a scalable generative model based on normalizing flows that achieves strong performance in high-resolution image synthesis. The core of STARFlow is Transformer Autoregressive Flow (TARFlow), which combines the expressive power of normalizing flows with the structured modeling capabilities of Autoregressive Transformers. We first establish the theoretical universality of TARFlow for modeling continuous distributions. Building on this foundation, we introduce several key architectural and algorithmic innovations to significantly enhance scalability: (1) a deep-shallow design, wherein a deep Transformer block captures most of the model representational capacity, complemented by a few shallow Transformer blocks that are computationally efficient yet substantially beneficial; (2) modeling in the latent space of pretrained autoencoders, which proves more effective than direct pixel-level modeling; and (3) a novel guidance algorithm that significantly boosts sample quality. Crucially, our model remains an end-to-end normalizing flow, enabling exact maximum likelihood training in continuous spaces without discretization. STARFlow achieves competitive performance in both class-conditional and text-conditional image generation tasks, approaching state-of-the-art diffusion models in sample quality. To our knowledge, this work is the first successful demonstration of normalizing flows operating effectively at this scale and resolution.

Figures (17)

• Figure 1: Text conditioned high-resolution samples of variable aspect ratios generated from our $3.8$B STARFlow model. Resolutions are adjusted for the ease of visualization.
• Figure 2: Ex. of 2-block AFs.
• Figure 3: Top to bottom, guiding the first 0, 3, 8 flow blocks with a TARFlow model with 8 flow blocks. We see that guidance is only effective up to the top 3 blocks.
• Figure 4: An illustration of the autoregressive inference (left) and parallel training (right) process of our proposed model for text-to-image generation. The upward (green) and downward (purple) arrows refers to the inverse and forward AF step as shown in \ref{['eq.af_step']}.
• Figure 5: (a) Guidance from TARFlow zhai2024normalizing (b) Proposed guidance on ImageNet 256 $\times$ 256.

Theorems & Definitions (1)

• proof