Unleashing the Potential of Large Language Models for Text-to-Image Generation through Autoregressive Representation Alignment

TL;DR

The paper tackles the challenge of achieving global, spatially coherent text-to-image generation with autoregressive LLMs without modifying their architecture. It introduces Autoregressive Representation Alignment (ARRA), which couples the standard next-token loss with a Global Visual Alignment loss that distills external visual representations into a novel <HYBNEXT> token during training. ARRA comes in three practical variants (ARRA-Base, ARRA, ARRA-Adapt) to train from scratch, convert text-generation-only LLMs, or adapt to domain-specific settings, respectively. Across natural and medical imaging tasks, ARRA substantially improves FID and CLIP-Score, demonstrating that objective redesign, rather than architectural changes, can resolve cross-modal coherence gaps while preserving inference efficiency. The work offers a versatile, plug-and-play framework with strong implications for scalable, coherent multimodal generation in autoregressive models.

Abstract

We present Autoregressive Representation Alignment (ARRA), a new training framework that unlocks global-coherent text-to-image generation in autoregressive LLMs without architectural modifications. Different from prior works that require complex architectural redesigns, ARRA aligns LLM's hidden states with visual representations from external visual foundational models via a global visual alignment loss and a hybrid token, <HYBNEXT>. This token enforces dual constraints: local next-token prediction and global semantic distillation, enabling LLMs to implicitly learn spatial and contextual coherence while retaining their original autoregressive paradigm. Extensive experiments validate ARRA's plug-and-play versatility. When training T2I LLMs from scratch, ARRA reduces FID by 16.6% (ImageNet), 12.0% (LAION-COCO) for autoregressive LLMs like LlamaGen, without modifying original architecture and inference mechanism. For training from text-generation-only LLMs, ARRA reduces FID by 25.5% (MIMIC-CXR), 8.8% (DeepEyeNet) for advanced LLMs like Chameleon. For domain adaptation, ARRA aligns general-purpose LLMs with specialized models (e.g., BioMedCLIP), achieving an 18.6% FID reduction over direct fine-tuning on medical imaging (MIMIC-CXR). These results demonstrate that training objective redesign, rather than architectural modifications, can resolve cross-modal global coherence challenges. ARRA offers a complementary paradigm for advancing autoregressive models. The code is available at https://github.com/HKU-HealthAI/ARRA.

Figures (13)

• Figure 1: ARRA enables high-quality text-to-image generation through a redefined training objective that promotes global coherence. (a)(c) Traditional next-token prediction (NTP)-based LLMs rely solely on the autoregressive loss (AR loss) of the next token $\texttt{<NEXT>}$ for local constraints. (b)(d) ARRA constructs a hybrid token $\texttt{<HYBNEXT>}$, which is aligned by introducing external global visual representation, ensuring that $\texttt{<HYBNEXT>}$ is constrained both locally by the AR loss and globally by the global visual alignment loss (GVA loss). (e)(f) ARRA demonstrates advantages in semantic consistency and visual continuity.
• Figure 1: More visualizations on the chest X-ray generation task across the MIMIC-CXR datasets.
• Figure 2: Proposed ARRA Framework. We define the next token predicted in the autoregressive sequence as the "HYBRID next token", denoted as $\texttt{<HYBNEXT>}$. During training, $\texttt{<HYBNEXT>}$ is constrained not only locally by the autoregressive loss $\mathcal{L}_{\text{AR}}$ from the next token prediction and LLM codebook matching, but also globally through visual alignment loss $\mathcal{L}_{\text{GVA}}$, which modulates its hidden states using externally well-trained representations. We extract visual representations from a pretrained foundational model and further aggregate these features to obtain semantically enriched representations for alignment.
• Figure 2: Visualization of alignment with features extracted from different encoders.
• Figure 3: ARRA-Base improves the generation of LlamaGen. Left are text-conditional image generation results on the LAION-COCO dataset, and right are category-conditional image generation results on the Imagenet dataset.