Efficient Conditional Generation on Scale-based Visual Autoregressive Models

TL;DR

This work introduces ECM, a plug-and-play, adapter-based framework for efficient spatial conditional generation in scale-based autoregressive image models. By distributing lightweight adapters across a frozen pre-trained backbone and employing early-centric training plus temperature scheduling, ECM achieves high-fidelity, diverse conditional outputs with substantially reduced training and inference costs. Key innovations include partial adapter sharing with layer-specific gating and an early-token bias that concentrates learning on foundational structure. Across ImageNet-1k experiments, ECM outperforms strong baselines like ControlVAR while using a fraction of their parameter budget and training time, demonstrating practical gains for conditional generation at scale.

Abstract

Recent advances in autoregressive (AR) models have demonstrated their potential to rival diffusion models in image synthesis. However, for complex spatially-conditioned generation, current AR approaches rely on fine-tuning the pre-trained model, leading to significant training costs. In this paper, we propose the Efficient Control Model (ECM), a plug-and-play framework featuring a lightweight control module that introduces control signals via a distributed architecture. This architecture consists of context-aware attention layers that refine conditional features using real-time generated tokens, and a shared gated feed-forward network (FFN) designed to maximize the utilization of its limited capacity and ensure coherent control feature learning. Furthermore, recognizing the critical role of early-stage generation in determining semantic structure, we introduce an early-centric sampling strategy that prioritizes learning early control sequences. This approach reduces computational cost by lowering the number of training tokens per iteration, while a complementary temperature scheduling during inference compensates for the resulting insufficient training of late-stage tokens. Extensive experiments on scale-based AR models validate that our method achieves high-fidelity and diverse control over image generation, surpassing existing baselines while significantly improving both training and inference efficiency.

Figures (16)

• Figure 1: Visualization of ECM's conditional generation. We leverage only a 300M-parameter model to achieve high-quality conditional image synthesis at 256$\times$256 resolution.
• Figure 2: VAR performs 10-step AR generation for 256×256 resolution images. We inject control signals during the first and final three steps. Empirical results show that early control injection effectively guides the generation process, while late injection confers minimal control effect and risks compromising output quality.
• Figure 3: Workflow and architecture of ECM. On the right, the ECM architecture features multiple adapter blocks distributed evenly throughout the network. Each adapter fuses image and control tokens using element-wise addition, generating adaptive control signals. These signals are processed by a shared FFN that promotes coherent control pattern learning, while its internal layer-specific gating instills positional awareness in each adapter block. On the left, this architecture is supported by two complementary strategies: first, early-centric sampling prioritizes critical early control patterns during training for greater efficiency. Second, a temperature scheduling scheme is applied during inference, lowering the temperature for later tokens to compensate for the reduced training focus and maintain high-quality output.
• Figure 4: Analysis of attention from the early, middle, and late stages of VAR. The results reveal a transition, shifting from global structures (early stage) to localized features (late stage).
• Figure 5: Training time shows that our method achieves substantial reductions in training costs (w.o. and w. sampling means without and with early-centric sampling). For inference time (per-image generation time), the left part is conducted using batch size equals 5 and the right equals 1).