Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models

TL;DR

This work tackles the high token-count barrier of autoregressive image generation by introducing Token-Shuffle, a pair of token-fusion and -unfusion operations that exploit visual dimensional redundancy to dramatically reduce Transformer input tokens without sacrificing output detail. By combining discrete visual tokens with LLM-like next-token prediction and a CFG-based synthesis strategy, the method enables ultra-high-resolution image generation up to $2048\times 2048$ in an autoregressive framework. Empirical results on GenAI-Bench, GenEval, and human evaluations show Token-Shuffle achieving competitive or superior text alignment and visual quality compared to autoregressive and diffusion baselines, while offering substantial efficiency gains. The approach provides a foundational design for scalable, high-fidelity image generation within Multimodal Large Language Models, with clear avenues for scaling to larger models and exploring flexible resolutions.

Abstract

Autoregressive (AR) models, long dominant in language generation, are increasingly applied to image synthesis but are often considered less competitive than Diffusion-based models. A primary limitation is the substantial number of image tokens required for AR models, which constrains both training and inference efficiency, as well as image resolution. To address this, we present Token-Shuffle, a novel yet simple method that reduces the number of image tokens in Transformer. Our key insight is the dimensional redundancy of visual vocabularies in Multimodal Large Language Models (MLLMs), where low-dimensional visual codes from visual encoder are directly mapped to high-dimensional language vocabularies. Leveraging this, we consider two key operations: token-shuffle, which merges spatially local tokens along channel dimension to decrease the input token number, and token-unshuffle, which untangles the inferred tokens after Transformer blocks to restore the spatial arrangement for output. Jointly training with textual prompts, our strategy requires no additional pretrained text-encoder and enables MLLMs to support extremely high-resolution image synthesis in a unified next-token prediction way while maintaining efficient training and inference. For the first time, we push the boundary of AR text-to-image generation to a resolution of 2048x2048 with gratifying generation performance. In GenAI-benchmark, our 2.7B model achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen by 0.18 and diffusion models LDM by 0.15. Exhaustive large-scale human evaluations also demonstrate our prominent image generation ability in terms of text-alignment, visual flaw, and visual appearance. We hope that Token-Shuffle can serve as a foundational design for efficient high-resolution image generation within MLLMs.

Figures (21)

• Figure 1: High-resolution images generated by our 2.7B AR model with Token-Shuffle (shuffle window size = 2).
• Figure 2: Token-Shuffle Pipeline: a plug-and-play operation pair for reducing visual token number in MLLMs, comprising a token-shuffle operation to merge spatially local visual tokens for Transformer input and a token-unshuffle operation to disentangle inferred visual tokens.
• Figure 3: Illustration of visual vocabulary dimensional redundancy. Left: Two MLPs reduce visual token rank by a factor of $r$. Right: Pre-training loss (log-scaled perplexity) for different $r$ values, showing substantial dimension reduction with minimal performance impact.
• Figure 4: Token-Shuffle can enhance efficiency quadratically. For instance, with a shuffle window size $s=2$, we achieve approximately a $4\times$ reduction in both training FLOPs and token number. Considering the use of KV-cache during inference, inference time scales roughly linearly with the token number.
• Figure 5: Comparison of different CFG schedulers with a monotonic increase in CFG scale from 1 to 7.5. Right: CFG-scheduler improves both visual aesthetics and text alignment, compared to the baseline of a consistent CFG value of 7.5 across all visual tokens.