Next Patch Prediction for Autoregressive Visual Generation

TL;DR

The paper introduces Next Patch Prediction (NPP) to autoregressive image generation by aggregating image tokens into high-density patch tokens, reducing input sequence length and training cost. A multi-scale coarse-to-fine grouping strategy enables a seamless transition from patch-based predictions to vanilla Next Token Prediction, preserving inference and avoiding extra trainable parameters. Empirical results on ImageNet-256×256 demonstrate substantial efficiency gains (≈0.6× cost) and improved generation quality (up to ~1.0 FID point) across model sizes, outperforming strong baselines like LlamaGen. The approach offers a flexible, plug-and-play enhancement for autoregressive visual generation and paves the way for more unified language-vision modeling.

Abstract

Autoregressive models, built based on the Next Token Prediction (NTP) paradigm, show great potential in developing a unified framework that integrates both language and vision tasks. Pioneering works introduce NTP to autoregressive visual generation tasks. In this work, we rethink the NTP for autoregressive image generation and extend it to a novel Next Patch Prediction (NPP) paradigm. Our key idea is to group and aggregate image tokens into patch tokens with higher information density. By using patch tokens as a more compact input sequence, the autoregressive model is trained to predict the next patch, significantly reducing computational costs. To further exploit the natural hierarchical structure of image data, we propose a multi-scale coarse-to-fine patch grouping strategy. With this strategy, the training process begins with a large patch size and ends with vanilla NTP where the patch size is 1$\times$1, thus maintaining the original inference process without modifications. Extensive experiments across a diverse range of model sizes demonstrate that NPP could reduce the training cost to around 0.6 times while improving image generation quality by up to 1.0 FID score on the ImageNet 256x256 generation benchmark. Notably, our method retains the original autoregressive model architecture without introducing additional trainable parameters or specifically designing a custom image tokenizer, offering a flexible and plug-and-play solution for enhancing autoregressive visual generation.

Figures (6)

• Figure 1: Comparison of our method and baseline methods. Our method on a diverse range of models achieves higher FID scores with significantly less training cost on the ImageNet 256$\times$256 generation benchmark. Our method NPP-L achieves up to $4.0 \times$ training speed up without performance degradation compared to LlamaGen-L-384.
• Figure 2: Motivation of the next patch prediction. a). Illustration of next token prediction. b). Demonstration of the proposed next patch prediction. c). Generation results on the ImageNet benchmark. Please zoom in to view.
• Figure 3: Next Patch Prediction. The input image token embeddings are grouped and aggregated into patch embeddings through a path average operation. The autoregressive model is trained to predict the next patch by employing the patch Cross Entropy loss.
• Figure 4: Multi-scale Next Patch Prediction. The patch grouping function begins with a large patch size, resulting in a short sequence length. As training progresses, the patch size is gradually reduced to $1\times1$.
• Figure 5: Comparison of our method and baseline methods. The vertical axes are the FID score and IS score. We record the performance curve with the number of epochs as horizontal axes.