Elucidating the design space of language models for image generation

TL;DR

This work is the first to analyze the optimization behavior of language models in vision generation, and it is believed it can inspire more effective designs when applying LMs to other domains.

Abstract

The success of autoregressive (AR) language models in text generation has inspired the computer vision community to adopt Large Language Models (LLMs) for image generation. However, considering the essential differences between text and image modalities, the design space of language models for image generation remains underexplored. We observe that image tokens exhibit greater randomness compared to text tokens, which presents challenges when training with token prediction. Nevertheless, AR models demonstrate their potential by effectively learning patterns even from a seemingly suboptimal optimization problem. Our analysis also reveals that while all models successfully grasp the importance of local information in image generation, smaller models struggle to capture the global context. In contrast, larger models showcase improved capabilities in this area, helping to explain the performance gains achieved when scaling up model size. We further elucidate the design space of language models for vision generation, including tokenizer choice, model choice, model scalability, vocabulary design, and sampling strategy through extensive comparative experiments. Our work is the first to analyze the optimization behavior of language models in vision generation, and we believe it can inspire more effective designs when applying LMs to other domains. Finally, our elucidated language model for image generation, termed as ELM, achieves state-of-the-art performance on the ImageNet 256*256 benchmark. The code is available at https://github.com/Pepperlll/LMforImageGeneration.git.

Figures (16)

• Figure 1: Generated samples from ELM-2B with 2-12 tokenizer trained on 256$\times$256 ImageNet. ELM is flexible to generate any-size high-fidelity images.
• Figure 2: BAE-16 exhibits a higher code utilization than VQGAN-f16. This figure shows a log count number of the appearance of codes on the ImageNet training dataset in sorted order. (a) BAE-16, with a code dimension of 16, has 65,536 unique codes and achieves 100% code utilization, with no code showing extremely low usage. In contrast, (b) VQGAN-f16, with a codebook size of 16,384, only utilizes around 1,000 codes, and many of these codes have extremely low utilization.
• Figure 3: Comparison of AR and MLM on image generation with 50,000 generated samples. AR consistently outperforms MLM across various model sizes.
• Figure 4: AR model performance with different BAE tokenizers.
• Figure 5: The results of different $\tau$ with MLM.