StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets

TL;DR

This work tackles the challenge of scaling high-fidelity generative models to large, unstructured datasets like ImageNet by rethinking training strategy rather than just architecture. It combines Projected GAN training, progressive growing, multiple pretrained feature networks, and classifier guidance to scale StyleGAN3 to $1024^2$ and ImageNet-scale synthesis, achieving state-of-the-art results. Beyond generation, StyleGAN-XL demonstrates robust inversion and editing capabilities, aided by PTI and latent-space manipulations, while offering significant speed advantages over diffusion models. The approach establishes a practical pathway for high-resolution, diverse, editable image synthesis on very large datasets and provides open-source tooling for further research.

Abstract

Computer graphics has experienced a recent surge of data-centric approaches for photorealistic and controllable content creation. StyleGAN in particular sets new standards for generative modeling regarding image quality and controllability. However, StyleGAN's performance severely degrades on large unstructured datasets such as ImageNet. StyleGAN was designed for controllability; hence, prior works suspect its restrictive design to be unsuitable for diverse datasets. In contrast, we find the main limiting factor to be the current training strategy. Following the recently introduced Projected GAN paradigm, we leverage powerful neural network priors and a progressive growing strategy to successfully train the latest StyleGAN3 generator on ImageNet. Our final model, StyleGAN-XL, sets a new state-of-the-art on large-scale image synthesis and is the first to generate images at a resolution of $1024^2$ at such a dataset scale. We demonstrate that this model can invert and edit images beyond the narrow domain of portraits or specific object classes.

Figures (20)

• Figure 1: Training StyleGAN-XL. We feed a latent code $\mathbf{z}$ and class label $\mathbf{c}$ to the pretrained embedding and the mapping network $\mathbf{G}_m$ to generate style codes $\mathbf{w}$. The codes modulate the convolutions of the synthesis network $\mathbf{G}_s$. During training, we gradually add layers to double the output resolution for each stage of the progressive growing schedule. We only train the latest layers while keeping the others fixed. $\mathbf{G}_m$ is only trained for the initial $16^2$ stage and remains fixed for the higher-resolution stages. The synthesized image is upsampled when smaller than $224^2$ and passed through a CNN and a ViT and respective feature mixing blocks (CCM+CSM). At higher resolutions, the CNN receives the unaltered image while the ViT receives a downsampled input to keep memory requirements low but still utilize its global feedback. Finally, we apply eight independent discriminators on the resulting multi-scale feature maps. The image is also fed to classifier CLF for classifier guidance.
• Figure 1: Ablation Study on ImageNet $128^2$. Left: Results for different configurations after training for $15$ V100-days. Right: Comparing combinations of different feature networks $\mathbf{F}$. Beginning from the base configuration using an EfficientNet-lite0 (EffNet), we add a second $\mathbf{F}$ with varying architecture type and pretraining objective (Class: Classification, Self: MoCo-v2 Chen2020ARXIV).
• Figure 2: Image Synthesis on ImageNet. Empty cells indicate that the model was not available and the respective metric not evaluated in the original work.
• Figure 2: Samples at Different Resolutions Using the Same $\mathbf{w}$. The samples are generated by the models obtained during progressive growing. We upsample all images to $1024^2$ using nearest-neighbor interpolation for visualization purposes. Zooming in is recommended.
• Figure 3: Interpolations. StyleGAN-XL generates smooth interpolations between samples of different classes (Row 1 & Row 2). PTI allows inverting to the latent space with low distortion (outermost image, Row 3 & Row 4), and consistently embeds out-of-domain inputs, such as the one on the bottom right.