Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nexus-Gen: Unified Image Understanding, Generation, and Editing via Prefilled Autoregression in Shared Embedding Space

Hong Zhang, Zhongjie Duan, Xingjun Wang, Yuze Zhao, Weiyi Lu, Zhipeng Di, Yixuan Xu, Yingda Chen, Yu Zhang

TL;DR

Nexus-Gen tackles the fragmentation of image understanding, generation, and editing by introducing a unified image embedding space that bridges LLM-based reasoning and diffusion-based image synthesis. A novel prefilled autoregression strategy aligns training and inference to mitigate error accumulation in continuous embeddings, complemented by a three-stage multi-task training scheme on a 26.3M dataset and a high-quality ImagePulse editing set. The model achieves state-of-the-art results across image understanding, generation, and editing benchmarks, and its modular design enables efficient cross-task reuse of embeddings. This work advances multimodal learning by enabling joint optimization and cross-turn reasoning, with practical implications for unified multimodal systems and editing pipelines.

Abstract

Unified multimodal generative models aim to integrate image understanding and generation abilities, offering significant advantages in harnessing multimodal corpora, particularly interleaved text-image data. However, existing unified models exhibit limitations in image synthesis quality, autoregressive error accumulation, and image editing capability. In this work, we propose Nexus-Gen, a novel architecture that unifies image understanding, generation, and editing tasks in a shared image embedding space. This shared space serves as a bridge for the autoregressive and diffusion models, which seamlessly integrates their complementary strengths in cross-modal modeling. To mitigate the severe error accumulation during autoregressive embedding prediction, we propose a novel prefilled autoregression strategy that aligns training-inference dynamics by prefilling input sequences with learnable embeddings. After multi-stage and multi-task training on our constructed large-scale dataset with 26.3 million samples, Nexus-Gen achieves state-of-the-art performance on the evaluation benchmarks spanning image understanding, generation and editing tasks. All models, datasets, and source codes are released in https://github.com/modelscope/Nexus-Gen to facilitate further advancements across the field.

Nexus-Gen: Unified Image Understanding, Generation, and Editing via Prefilled Autoregression in Shared Embedding Space

TL;DR

Nexus-Gen tackles the fragmentation of image understanding, generation, and editing by introducing a unified image embedding space that bridges LLM-based reasoning and diffusion-based image synthesis. A novel prefilled autoregression strategy aligns training and inference to mitigate error accumulation in continuous embeddings, complemented by a three-stage multi-task training scheme on a 26.3M dataset and a high-quality ImagePulse editing set. The model achieves state-of-the-art results across image understanding, generation, and editing benchmarks, and its modular design enables efficient cross-task reuse of embeddings. This work advances multimodal learning by enabling joint optimization and cross-turn reasoning, with practical implications for unified multimodal systems and editing pipelines.

Abstract

Unified multimodal generative models aim to integrate image understanding and generation abilities, offering significant advantages in harnessing multimodal corpora, particularly interleaved text-image data. However, existing unified models exhibit limitations in image synthesis quality, autoregressive error accumulation, and image editing capability. In this work, we propose Nexus-Gen, a novel architecture that unifies image understanding, generation, and editing tasks in a shared image embedding space. This shared space serves as a bridge for the autoregressive and diffusion models, which seamlessly integrates their complementary strengths in cross-modal modeling. To mitigate the severe error accumulation during autoregressive embedding prediction, we propose a novel prefilled autoregression strategy that aligns training-inference dynamics by prefilling input sequences with learnable embeddings. After multi-stage and multi-task training on our constructed large-scale dataset with 26.3 million samples, Nexus-Gen achieves state-of-the-art performance on the evaluation benchmarks spanning image understanding, generation and editing tasks. All models, datasets, and source codes are released in https://github.com/modelscope/Nexus-Gen to facilitate further advancements across the field.
Paper Structure (42 sections, 6 equations, 10 figures, 4 tables)

This paper contains 42 sections, 6 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Autoregressive Model with VAE
  4. Autoregressive Model with Diffusion
  5. Approach
  6. Architecture
  7. Unified Image Embedding Space
  8. Vision Encoder
  9. Autoregressive Model
  10. Vision Decoder
  11. Prefilled Autoregression
  12. Dataset Curation
  13. Training Objectives
  14. Training Strategy
  15. Multi-Task Pretraining for Autoregressive Model
  16. ...and 27 more sections

Figures (10)

  • Figure 1: The architecture and the multi-stage training recipe for Nexus-Gen.
  • Figure 2: (a) The naive autoregressive approach exhibits inconsistent behaviors between training and test phases, leading to error accumulation during inference. (b) We propose a novel strategy that prefills special image tokens during training and testing, which unifies the computational behaviors across both phases and eliminates error accumulation.
  • Figure 3: Image reconstruction results of our generation decoder using 81 and 324 image token embedding.
  • Figure 4: Image generation results from Nexus-Gen trained with 324 and 81 image token embeddings.
  • Figure 5: Image editing results of Nexus-Gen with editing and generation decoder.
  • ...and 5 more figures