Scaling Law Hypothesis for Multimodal Model

TL;DR

This work proposes a scaling law hypothesis for multimodal models processing text, audio, images, and video within a shared token and embedding space and explores whether leveraging more training data in multiple modalities can reduce the size of the multimodal model, enabling efficient deployment on resource-constrained devices.

Abstract

We propose a scaling law hypothesis for multimodal models processing text, audio, images, and video within a shared token and embedding space. Our framework predicts model performance based on modality-specific compression and tokenization efficiency, extending established scaling laws from text-based decoder models to mixed-modality systems. We explore whether leveraging more training data in multiple modalities can reduce the size of the multimodal model, enabling efficient deployment on resource-constrained devices.

Figures (2)

• Figure 1: Performance (measured by BPC) correlates with data compression ability across different text corpora.
• Figure 2: Performance scales linearly with compute when measured by BPC guo2024computeneed.