Heptapod: Language Modeling on Visual Signals

TL;DR

Heptapod rethinks vision-language modeling by extending autoregressive language modeling principles to 2D visual data. It introduces next 2D distribution prediction, where a causal Transformer predicts distributions over the entire 2D image grid at each timestep, using a reconstruction-focused tokenizer to avoid external semantic injections. The approach achieves state-of-the-art results among causal autoregressive models on ImageNet generation without CFG, and ablations show the importance of global 2D prediction and tokenization fidelity in learning holistic image semantics. By decoupling tokenization from semantic learning and unifying autoregressive and MAE-style objectives, Heptapod lays groundwork for principled visual language modeling and potential cross-modal extensions.

Abstract

We introduce Heptapod, an image autoregressive model that adheres to the foundational principles of language modeling. Heptapod employs \textbf{causal attention}, \textbf{eliminates reliance on CFG}, and \textbf{eschews the trend of semantic tokenizers}. Our key innovation is \textit{next 2D distribution prediction}: a causal Transformer with reconstruction-focused visual tokenizer, learns to predict the distribution over the entire 2D spatial grid of images at each timestep. This learning objective unifies the sequential modeling of autoregressive framework with the holistic self-supervised learning of masked autoencoding, enabling the model to capture comprehensive image semantics via generative training. On the ImageNet generation benchmark, Heptapod achieves an FID of $2.70$, significantly outperforming previous causal autoregressive approaches. We hope our work inspires a principled rethinking of language modeling on visual signals and beyond.

Figures (13)

• Figure 1: (Left) A typical visual latent generative framework that incorporates external semantics. (Top Right) Leading autoregressive models exhibit steep performance drops when CFG is disabled (VAR results from chen2025toward). (Bottom Right) Generation quality degrades when the external SSL model DINO is removed (CFG is disabled).
• Figure 2: Illustration of Heptapod's next 2D distribution prediction framework. The model operates on a sequence of visual tokens from a simple reconstruction-focused tokenizer. The Transformer autoregressively predicts the distributions over remaining positions in the 2D grid in parallel for every input tokens. The loss is then computed across all these future positions, treating the prefix as the visible context (like MAE's unmasked patches) and the remaining grid as targets. This forces the model to develop a holistic representation, bridging the gap between 1D language modeling and 2D spatial understanding.
• Figure 3: (Left) Spatial correlations in VQ-VAE vs. semantic tokenizers. For three reference tokens (87th, 138th and 203rd, bounded by red lines), we compute cosine similarity to all other tokens in the grid. (Right) Attention maps of the final layer in autoregressive Transformer trained with each tokenizer. Under VQ-VAE, attention concentrates on spatial neighbors (local interpolation), while semantic tokens yield attention on spatially distant yet semantically related regions (long-range dependencies). Following DiGIT NEURIPS2024_325ce329, semantic tokens are obtained by K-Means on DINO hidden states. Additional examples are provided in Appendix \ref{['appendix:similarity_grid']}.
• Figure 4: (Left) The text language modeling with next 1D distribution prediction. (Middle) Vanilla image language modeling with next 1D distribution prediction. The 2D sptial postion is left-shifted in the input to specify the next target. (Right) Our next 2D distribution prediction. The 2D spatial positions are not shifted. The model must be prepared to predict any future position.
• Figure 5: Architectural variants of the 2D prediction head. (Left) The global prediction head employs a bidirectional transformer to model full-image spatial dependencies. (Right) The local prediction head stacks cross attention and bidirectional attention to capture 2D spatial dependencies within a local chunk.