DPAR: Dynamic Patchification for Efficient Autoregressive Visual Generation

TL;DR

Decoder-only autoregressive image generation suffers from token counts that scale quadratically with image resolution. DPAR tackles this by entropy-guided dynamic patchification, merging low-information tokens into patches and letting a patch-level transformer generate images with fewer but information-rich units; a lightweight entropy model drives the patching, while a patch encoder/decoder preserve 2D structure. Empirically, DPAR reduces token counts by up to $2.06\times$ and training FLOPs by up to $40\%$, while achieving up to a $27.1\%$ relative improvement in FID on ImageNet benchmarks and faster convergence. The approach maintains compatibility with existing multimodal generation pipelines and yields representations robust to patch boundaries, enabling adaptive patch sizes at inference.

Abstract

Decoder-only autoregressive image generation typically relies on fixed-length tokenization schemes whose token counts grow quadratically with resolution, substantially increasing the computational and memory demands of attention. We present DPAR, a novel decoder-only autoregressive model that dynamically aggregates image tokens into a variable number of patches for efficient image generation. Our work is the first to demonstrate that next-token prediction entropy from a lightweight and unsupervised autoregressive model provides a reliable criterion for merging tokens into larger patches based on information content. DPAR makes minimal modifications to the standard decoder architecture, ensuring compatibility with multimodal generation frameworks and allocating more compute to generation of high-information image regions. Further, we demonstrate that training with dynamically sized patches yields representations that are robust to patch boundaries, allowing DPAR to scale to larger patch sizes at inference. DPAR reduces token count by 1.81x and 2.06x on Imagenet 256 and 384 generation resolution respectively, leading to a reduction of up to 40% FLOPs in training costs. Further, our method exhibits faster convergence and improves FID by up to 27.1% relative to baseline models.

Figures (12)

• Figure 1: Autoregressive image generation with DPAR. (a) We show selected samples from our class-conditional DPAR-384-XL model trained on ImageNet. (b) FID vs FLOPs comparison of DPAR model variants trained on 256x256 and 384x384 Image resolution on ImageNet. DPAR achieves reductions in training FLOPs by upto 40% while improving FID by up to 27.1% relative to baseline models.
• Figure 2: Images (first row) and their corresponding next-token prediction entropy maps (second row) with increasing information content. Images with lower information content produce fewer high-entropy tokens, allowing the model to merge them into larger patches for efficient AR generation. Entropy heatmaps are computed over 256 tokens for 256$\times$256 images, with black outlines indicating the final patch boundaries.
• Figure 3: Overview of DPAR. (a) Conventional AR image generation employs decoder-only transformers operating on a fixed number of tokens per image, where the token count increases quadratically with image resolution. (b) DPAR dynamically aggregates image tokens based on information content, generating a variable number of patches per image. Decoder-only transformers then operate on a smaller number of patches, reducing computational and memory overhead. DPAR makes minimal modifications to the standard decoder architecture, ensuring compatibility with multimodal generation frameworks.
• Figure 4: Comparative analysis of converge of DPAR with LLamaGen on ImageNet-384. We plot FID vs training epochs for various model sizes. DPAR consistently achieves lower FID scores, demonstrating faster convergence and better image fidelity.
• Figure 5: Uncurated generated samples for model DPAR-XL trained at 256$\times$256 resolution at CFG-scale=1.75