ScaleWeaver: Weaving Efficient Controllable T2I Generation with Multi-Scale Reference Attention

TL;DR

Extensive experiments show that ScaleWeaver delivers high-quality generation and precise control while attaining superior efficiency over diffusion-based methods, making ScaleWeaver a practical and effective solution for controllable text-to-image generation within the visual autoregressive paradigm.

Abstract

Text-to-image generation with visual autoregressive~(VAR) models has recently achieved impressive advances in generation fidelity and inference efficiency. While control mechanisms have been explored for diffusion models, enabling precise and flexible control within VAR paradigm remains underexplored. To bridge this critical gap, in this paper, we introduce ScaleWeaver, a novel framework designed to achieve high-fidelity, controllable generation upon advanced VAR models through parameter-efficient fine-tuning. The core module in ScaleWeaver is the improved MMDiT block with the proposed Reference Attention module, which efficiently and effectively incorporates conditional information. Different from MM Attention, the proposed Reference Attention module discards the unnecessary attention from image$\rightarrow$condition, reducing computational cost while stabilizing control injection. Besides, it strategically emphasizes parameter reuse, leveraging the capability of the VAR backbone itself with a few introduced parameters to process control information, and equipping a zero-initialized linear projection to ensure that control signals are incorporated effectively without disrupting the generative capability of the base model. Extensive experiments show that ScaleWeaver delivers high-quality generation and precise control while attaining superior efficiency over diffusion-based methods, making ScaleWeaver a practical and effective solution for controllable text-to-image generation within the visual autoregressive paradigm. Code and models will be released.

Figures (9)

• Figure 1: Images generated by ScaleWeaver. Our ScaleWeaver enables efficient and precise controllable text-to-image generation based on the visual autoregressive model, supporting diverse condition signals and producing high-fidelity images with strong text alignment and controllability.
• Figure 2: Illustration of ScaleWeaver framework. The conditional input is tokenized using the same multi-scale tokenizer as the image, processed by a LoRA-based conditional branch, and fused with image tokens via Reference Attention at each scale. In the Reference Attention module, image-side projections are frozen while the conditional branch is LoRA-tuned. Image queries attend to condition keys/values through cross-attention, gated by a zero-initialized projection to preserve base generation capability and enable gradual control. Unlike MM-Attention, Reference Attention removes the image$\rightarrow$condition path, reducing unnecessary computation and stabilizing control injection. For clarity, image-text cross-attention and FFN within the original Infinity blocks are omitted.
• Figure 3: Qualitative comparison of controllable text-to-image generation results. ScaleWeaver achieves high-fidelity synthesis and precise control across diverse conditions, outperforming diffusion-based baselines in both visual quality and controllability. Best viewed via zoom in.
• Figure 4: Diverse generation with the same sketch condition but different text prompts.
• Figure 5: Visualization comparison for ablation study. We show qualitative results for different injection mechanisms.