Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Language Modeling: An Exploration of Multimodal Pretraining

Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan Ghazvininejad, Mike Lewis, Nicolas Ballas, Amir Bar, Michael Rabbat, Jakob Verbeek, Luke Zettlemoyer, Koustuv Sinha, Yann LeCun, Saining Xie

TL;DR

It is demonstrated that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.

Abstract

The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.

Beyond Language Modeling: An Exploration of Multimodal Pretraining

TL;DR

It is demonstrated that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.

Abstract

The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.
Paper Structure (105 sections, 8 equations, 35 figures, 6 tables)

This paper contains 105 sections, 8 equations, 35 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Experiment Setup
  3. Training and Inference
  4. Language modeling.
  5. Image-wise flow matching for visual tokens.
  6. Mixture training.
  7. Causal vision and language masking.
  8. Inference.
  9. Model and Tokenization
  10. Unified decoder-only backbone.
  11. Text tokens.
  12. Visual tokens.
  13. Data Sources
  14. Evaluation
  15. Visual Representations for Unified Multimodal Pretraining
  16. ...and 90 more sections

Figures (35)

  • Figure 1: Overview of our study.Top: High-level model architecture. We train a single autoregressive model with next-token prediction for text and next-visual state prediction with flow matching. Bottom: We study five axes: visual representations, data, world modeling, architecture, and scaling.
  • Figure 2: Examples of training data. We train on diverse data including text, videos, image-text, and action-conditioned video.
  • Figure 3: Shared $\rightarrow$ Modality-specific FFN. Modality separation improves multimodal learning over shared FFN. We explore MoE design in \ref{['sec:moe']}.
  • Figure 4: RAE outperforms VAEs for both generation and understanding. RAE (SigLIP 2) achieves the best performance on DPGBench, GenEval, and VQA while maintaining text perplexity comparable to the text-only baseline.
  • Figure 5: Visual data does not compete with text. Text + Video matches the text-only PPL on DCLM, suggesting that raw visual data is compatible with language modeling.
  • ...and 30 more figures