Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modeling

TL;DR

This work tackles the challenge of tokenization-free, long-context byte-level language modeling by introducing the Multiscale Byte Language Model (MBLM), a hierarchical, model- and modality-agnostic decoder stack that can process up to $5{,}000{,}000$ bytes of context on a single GPU. By stacking autoregressive stage models—global decoders for cross-patch context and a local byte-level model for intra-patch generation—MBLM achieves subquadratic efficiency through input compression and gradient checkpointing, enabling training with full precision on very long sequences. Empirically, hybrid configurations combining Mamba global stages with Transformer local stages deliver strong performance and near-linear generation throughput, outperforming traditional Transformer baselines on long sequences, and the authors demonstrate, for the first time, byte-level visual question answering on CLEVR with competitive results without an encoder. The work highlights the potential of byte-level, multimodal foundation modeling and provides a flexible, open-source framework for future exploration into million-scale Byte sequence modeling and omnimodal applications.

Abstract

Bytes form the basis of the digital world and thus are a promising building block for multimodal foundation models. Recently, Byte Language Models (BLMs) have emerged to overcome tokenization, yet the excessive length of bytestreams requires new architectural paradigms. Therefore, we present the Multiscale Byte Language Model (MBLM), a model-agnostic hierarchical decoder stack that allows training with context windows of $5$M bytes on single GPU in full model precision. We thoroughly examine MBLM's performance with Transformer and Mamba blocks on both unimodal and multimodal tasks. Our experiments demonstrate that hybrid architectures are efficient in handling extremely long byte sequences during training while achieving near-linear generational efficiency. To the best of our knowledge, we present the first evaluation of BLMs on visual Q\&A tasks and find that, despite serializing images and the absence of an encoder, a MBLM with pure next token prediction can match custom CNN-LSTM architectures with designated classification heads. We show that MBLMs exhibit strong adaptability in integrating diverse data representations, including pixel and image filestream bytes, underlining their potential toward omnimodal foundation models. Source code is publicly available at: https://github.com/ai4sd/multiscale-byte-lm

Figures (13)

• Figure 1: The Multiscale Byte Language Model (MBLM) processes bytestreams from any modality that can be serialized into bytes. Each stage in the hierarchical architecture employs a decoder model to generate a new representation for input patches, which is subsequently passed to the next stage as augmented input. The final output of the MBLM is a bytestream formed by concatenating the outputs of the last stage, $n$.
• Figure 2: A 3D MBLM module with two global and one local decoder models and corresponding patch sizes $P_1=5, P_2=3, P_3=2$, operating on an input sequence $\mathbf{x}=\{x_0, x_2, \ldots, x_{29}\}$. Inputs to each stage are prepended with a trainable start token < S>. The updated patch representations of the input sequence output by the global models are added to the inputs of the next stage. The local model generates individual bytes, and the final outputs are concatenated.
• Figure 3: Each CLEVR image is annotated with multiple questions, e.g.: What is the shape of the tiny brown thing?$\rightarrow$Cylinder
• Figure 4: Training loss progression of a 3D MBLM with 350M parameters and a context window of 5 million bytes on a single GPU.
• Figure 5: Throughput expressed as time-per-byte for 2D and 3D MBLMs during training. Using Mamba as a local model on short patches results in a 4x slower backwards phase.