Progressive Checkerboards for Autoregressive Multiscale Image Generation

TL;DR

The paper tackles the challenge of efficient autoregressive image generation by introducing a balanced progressive checkerboard order that enables parallel sampling across multiple regions within a multiscale pyramid. It presents a multiscale blockwise transformer autoregressor that conditions on upsampled latents from the previous scale and current-scale outputs, organized via a balanced quadtree-based scan with P blocks per scale. Empirical results on ImageNet 256x256 show competitive FID/IS with relatively few sampling steps (approximately $16$–$17$ total), and the key finding is that the total number of sequential steps dominates performance more than the exact scale factor. Additional analyses reveal limited benefits from RoPE mixing and reveal entropy dynamics and checkerboard patterns that reflect the method’s local conditioning capabilities. The approach offers practical gains in sampling efficiency and opens avenues for extending multiscale autoregression to other modalities.

Abstract

A key challenge in autoregressive image generation is to efficiently sample independent locations in parallel, while still modeling mutual dependencies with serial conditioning. Some recent works have addressed this by conditioning between scales in a multiscale pyramid. Others have looked at parallelizing samples in a single image using regular partitions or randomized orders. In this work we examine a flexible, fixed ordering based on progressive checkerboards for multiscale autoregressive image generation. Our ordering draws samples in parallel from evenly spaced regions at each scale, maintaining full balance in all levels of a quadtree subdivision at each step. This enables effective conditioning both between and within scales. Intriguingly, we find evidence that in our balanced setting, a wide range of scale-up factors lead to similar results, so long as the total number of serial steps is constant. On class-conditional ImageNet, our method achieves competitive performance compared to recent state-of-the-art autoregressive systems with like model capacity, using fewer sampling steps.

Figures (8)

• Figure 1: Progressive checkerboard samples from our model using 2x scale factor and 8 steps per scale. Masking applied to sampled locations at each step after decoding for visualization.
• Figure 2: Overview of our multiscale blockwise checkerboard autoregressor. (For illustration there are $P=4$ checkerboard blocks, while our actual model uses $P=8$ for more spatial distancing.)
• Figure 3: (a) Progressive checkerboard on an $8\times8$ grid using $P=8$ steps. (b) Full ordering.
• Figure 4: (a) FID vs IS computed at 0.1 CFG increments. L model size at 2, 4 and 8 steps per scale. (b) Exact scale sizes used in our experiments, for 256x256 image size and 16x16 VAE patch size.
• Figure 5: FID (top) and IS (bottom), by scale ratio and number of inference steps for the S model size. Left: by numbers of steps per scale; Right: by total number of steps. Multiscale models outperform the single-scale baseline. Note that scale ratios 2, 3 and 4 all achieve similar performance counted by total steps, even though the steps per scale needed to reach each total is different for each scale factor.