Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Simple Baseline for Unifying Understanding, Generation, and Editing via Vanilla Next-token Prediction

Jie Zhu, Hanghang Ma, Jia Wang, Yayong Guan, Yanbing Zeng, Lishuai Gao, Junqiang Wu, Jie Hu, Leye Wang

TL;DR

Wallaroo is introduced, a simple autoregressive baseline that leverages next-token prediction to unify multi-modal understanding, image generation, and editing at the same time and supports multi-resolution image input and output, as well as bilingual support for both Chinese and English.

Abstract

In this work, we introduce Wallaroo, a simple autoregressive baseline that leverages next-token prediction to unify multi-modal understanding, image generation, and editing at the same time. Moreover, Wallaroo supports multi-resolution image input and output, as well as bilingual support for both Chinese and English. We decouple the visual encoding into separate pathways and apply a four-stage training strategy to reshape the model's capabilities. Experiments are conducted on various benchmarks where Wallaroo produces competitive performance or exceeds other unified models, suggesting the great potential of autoregressive models in unifying multi-modality understanding and generation. Our code is available at https://github.com/JiePKU/Wallaroo.

A Simple Baseline for Unifying Understanding, Generation, and Editing via Vanilla Next-token Prediction

TL;DR

Wallaroo is introduced, a simple autoregressive baseline that leverages next-token prediction to unify multi-modal understanding, image generation, and editing at the same time and supports multi-resolution image input and output, as well as bilingual support for both Chinese and English.

Abstract

In this work, we introduce Wallaroo, a simple autoregressive baseline that leverages next-token prediction to unify multi-modal understanding, image generation, and editing at the same time. Moreover, Wallaroo supports multi-resolution image input and output, as well as bilingual support for both Chinese and English. We decouple the visual encoding into separate pathways and apply a four-stage training strategy to reshape the model's capabilities. Experiments are conducted on various benchmarks where Wallaroo produces competitive performance or exceeds other unified models, suggesting the great potential of autoregressive models in unifying multi-modality understanding and generation. Our code is available at https://github.com/JiePKU/Wallaroo.
Paper Structure (18 sections, 1 equation, 3 figures, 8 tables)

This paper contains 18 sections, 1 equation, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Wallaroo
  4. Architecture
  5. Training Procedure
  6. Training Objective
  7. Inference
  8. Experiments
  9. Implementation Details
  10. Training Data
  11. Results
  12. Results on Multi-modal Understanding
  13. Results on Image Generation
  14. Results on Image Editing
  15. Ablation Studies
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Some text-to-image generation showcases of our Wallaroo.
  • Figure 2: Illustration of our Wallaroo. We decouple visual encoding into separate pathways for visual understanding and image generation. For editing, we integrate two complementary types of visual representations to improve Wallaroo's performance.
  • Figure 3: A four-stage training procedure of our Wallaroo based on Qwen2.5 VL. We use flame symbols to denote modules that update their parameters, and snowflake symbols to denote modules that keep their parameters fixed.