LINA: Linear Autoregressive Image Generative Models with Continuous Tokens

TL;DR

LINA tackles the efficiency bottleneck of autoregressive image generation with continuous tokens by thoroughly evaluating linear attention designs. It shows division-based normalization paired with a depthwise-convolution locality boost consistently scales better and improves performance across model sizes, while a novel KV gate enables flexible memory management in bidirectional attention. The resulting model delivers competitive 256×256 image quality and strong 1024×1024 text-to-image results, with about a 61% reduction in FLOPs per attention module compared to softmax-based approaches. Collectively, LINA demonstrates that carefully engineered linear attention can achieve high fidelity in long-sequence image generation, offering a practical and scalable alternative to full-softmax transformers in this domain.

Abstract

Autoregressive models with continuous tokens form a promising paradigm for visual generation, especially for text-to-image (T2I) synthesis, but they suffer from high computational cost. We study how to design compute-efficient linear attention within this framework. Specifically, we conduct a systematic empirical analysis of scaling behavior with respect to parameter counts under different design choices, focusing on (1) normalization paradigms in linear attention (division-based vs. subtraction-based) and (2) depthwise convolution for locality augmentation. Our results show that although subtraction-based normalization is effective for image classification, division-based normalization scales better for linear generative transformers. In addition, incorporating convolution for locality modeling plays a crucial role in autoregressive generation, consistent with findings in diffusion models. We further extend gating mechanisms, commonly used in causal linear attention, to the bidirectional setting and propose a KV gate. By introducing data-independent learnable parameters to the key and value states, the KV gate assigns token-wise memory weights, enabling flexible memory management similar to forget gates in language models. Based on these findings, we present LINA, a simple and compute-efficient T2I model built entirely on linear attention, capable of generating high-fidelity 1024x1024 images from user instructions. LINA achieves competitive performance on both class-conditional and T2I benchmarks, obtaining 2.18 FID on ImageNet (about 1.4B parameters) and 0.74 on GenEval (about 1.5B parameters). A single linear attention module reduces FLOPs by about 61 percent compared to softmax attention. Code and models are available at: https://github.com/techmonsterwang/LINA.

Figures (9)

• Figure 1: Qualitative results of 1024px samples powered by LINA.
• Figure 2: Overview of LINA: Fig. (a) illustrates the training pipeline, with a Connector for extracting text information, an Encoder to extract unmasked tokens, and a Decoder to reconstruct masked tokens for conditioning. A denoising flow matching MLP is used to sample tokens. Fig. (b) shows the division-based normalization linear attention, and our introduced DWC module and KV gate (Sec. \ref{['sec:method']}).
• Figure 3: DWC helps locality. (a) A random-order autoregressive model with bidirectional attention predicts next tokens based on the predicted tokens. When the target token (e.g., the 8th) is surrounded by predicted tokens (e.g., the 3rd), the model faces challenges due to the limited local modeling capacity. (b) DWC module gathers information from nearby known tokens when predicting the current token, thereby facilitating linear attention.
• Figure 4: Scaling behavior and KV gate results. Fig. (a) describes the class-conditional image generation results on the ImageNet 256$\times$256 benchmark using FID ($\downarrow$) and IS ($\uparrow$). Division-based linear attention with LAM achieves the best scaling performance. Detailed results are provided in Appendix \ref{['sec:appendix.scaling-results']}. Fig. (b) presents the learned KV gate of a 256px text-to-image LINA model.
• Figure 5: FLOPs comparison: a single module of linear attention vs. softmax attention. We use a batch size of 1, a sequence length of 5120, a hidden dimension of 1536, and 16 attention heads. Such configuration corresponds to how LINA-H operates at 1024px. Linear attention applies division-based normalization and incorporates both the DWC module and the KV gate. Compared with softmax attention, linear attention reduces FLOPs by $\sim$61%, showing computation efficiency.