Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale

NextStep Team, Chunrui Han, Guopeng Li, Jingwei Wu, Quan Sun, Yan Cai, Yuang Peng, Zheng Ge, Deyu Zhou, Haomiao Tang, Hongyu Zhou, Kenkun Liu, Ailin Huang, Bin Wang, Changxin Miao, Deshan Sun, En Yu, Fukun Yin, Gang Yu, Hao Nie, Haoran Lv, Hanpeng Hu, Jia Wang, Jian Zhou, Jianjian Sun, Kaijun Tan, Kang An, Kangheng Lin, Liang Zhao, Mei Chen, Peng Xing, Rui Wang, Shiyu Liu, Shutao Xia, Tianhao You, Wei Ji, Xianfang Zeng, Xin Han, Xuelin Zhang, Yana Wei, Yanming Xu, Yimin Jiang, Yingming Wang, Yu Zhou, Yucheng Han, Ziyang Meng, Binxing Jiao, Daxin Jiang, Xiangyu Zhang, Yibo Zhu

TL;DR

NextStep-1 presents a scalable autoregressive framework that unifies discrete text tokens and continuous image tokens through a flow-matching head, enabling high-fidelity image generation and competitive image editing at 14B parameters. The architecture couples a Flux VAE–based image tokenizer with a causal Transformer (initialized from Qwen2.5-14B), and trains on a richly diverse, multi-category data mix via a three-stage pretraining curriculum plus annealing, followed by SFT and DPO for alignment. Empirical results show strong text-to-image performance on alignment and world-knowledge benchmarks, as well as competitive editing capabilities, outperforming many autoregressive peers and approaching diffusion-model quality in some settings. The paper also analyzes the central role of tokenization, normalization under guidance, and the latent-space regularization in achieving stable generation, while candidly outlining artifacts, latency, and high-resolution training challenges as future directions.

Abstract

Prevailing autoregressive (AR) models for text-to-image generation either rely on heavy, computationally-intensive diffusion models to process continuous image tokens, or employ vector quantization (VQ) to obtain discrete tokens with quantization loss. In this paper, we push the autoregressive paradigm forward with NextStep-1, a 14B autoregressive model paired with a 157M flow matching head, training on discrete text tokens and continuous image tokens with next-token prediction objectives. NextStep-1 achieves state-of-the-art performance for autoregressive models in text-to-image generation tasks, exhibiting strong capabilities in high-fidelity image synthesis. Furthermore, our method shows strong performance in image editing, highlighting the power and versatility of our unified approach. To facilitate open research, we will release our code and models to the community.

NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale

TL;DR

NextStep-1 presents a scalable autoregressive framework that unifies discrete text tokens and continuous image tokens through a flow-matching head, enabling high-fidelity image generation and competitive image editing at 14B parameters. The architecture couples a Flux VAE–based image tokenizer with a causal Transformer (initialized from Qwen2.5-14B), and trains on a richly diverse, multi-category data mix via a three-stage pretraining curriculum plus annealing, followed by SFT and DPO for alignment. Empirical results show strong text-to-image performance on alignment and world-knowledge benchmarks, as well as competitive editing capabilities, outperforming many autoregressive peers and approaching diffusion-model quality in some settings. The paper also analyzes the central role of tokenization, normalization under guidance, and the latent-space regularization in achieving stable generation, while candidly outlining artifacts, latency, and high-resolution training challenges as future directions.

Abstract

Prevailing autoregressive (AR) models for text-to-image generation either rely on heavy, computationally-intensive diffusion models to process continuous image tokens, or employ vector quantization (VQ) to obtain discrete tokens with quantization loss. In this paper, we push the autoregressive paradigm forward with NextStep-1, a 14B autoregressive model paired with a 157M flow matching head, training on discrete text tokens and continuous image tokens with next-token prediction objectives. NextStep-1 achieves state-of-the-art performance for autoregressive models in text-to-image generation tasks, exhibiting strong capabilities in high-fidelity image synthesis. Furthermore, our method shows strong performance in image editing, highlighting the power and versatility of our unified approach. To facilitate open research, we will release our code and models to the community.

Paper Structure

This paper contains 41 sections, 5 equations, 8 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Framework
  3. Unified Multi-model Generation with Continuous Visual Tokens
  4. Model Architecture
  5. Image Tokenizer.
  6. Causal Transformer.
  7. Data
  8. Text-only Corpus
  9. Image-Text Pair Data
  10. Instruction-Guided Image-to-Image Data
  11. Interleaved data
  12. Training Recipe
  13. Training Image Tokenizer
  14. Pre-Training
  15. Stage1.
  16. ...and 26 more sections

Figures (8)

  • Figure 1: Overview of NextStep-1 in high-fidelity image generation, diverse image editing, and complex free-form manipulation.
  • Figure 2: Overview of NextStep-1 Framework. NextStep-1 employs a causal transformer to process tokenized text and image tokens. During training, Flow Matching Head predicts the continuous flow from a noise sample to the next target image patch, conditioned on the output hidden state. At inference, this allows for generating images by iteratively guiding noise to create the next patch.
  • Figure 3: Data processing of character-centric data.
  • Figure 4: Images generated under different flow-matching heads.
  • Figure 5: Evolution of per-token mean and variance over sampling steps under two CFG settings. At CFG = 1.5, the mean and variance stay close to 0 and 1, respectively, indicating stability. At CFG = 3.0, they drift significantly, causing image quality degradation. With normalization, the distributions of output latents remain stable across all CFG settings.
  • ...and 3 more figures