Visual Autoregressive Models Beat Diffusion Models on Inference Time Scaling

TL;DR

This work demonstrates that inference-time searching is more effective for discrete visual autoregressive models than for continuous diffusion models. By leveraging Infinity's next-scale prediction across 13 scales and applying beam search guided by diverse verifiers (ImageReward, CLIPScore, Aesthetic Score, and LLaVA-OneVision), a 2B parameter model can surpass a 12B diffusion model on standard benchmarks while using far fewer function evaluations. The results show strong gains on DrawBench and compositional benchmarks, and a direct comparison with diffusion models confirms the architectural suitability of discrete search for image generation. The study highlights that architecture, not just scale, governs efficiency and quality in inference-time image synthesis, suggesting that co-design of models and search strategies is a fruitful direction for practical, scalable visual generation.

Abstract

While inference-time scaling through search has revolutionized Large Language Models, translating these gains to image generation has proven difficult. Recent attempts to apply search strategies to continuous diffusion models show limited benefits, with simple random sampling often performing best. We demonstrate that the discrete, sequential nature of visual autoregressive models enables effective search for image generation. We show that beam search substantially improves text-to-image generation, enabling a 2B parameter autoregressive model to outperform a 12B parameter diffusion model across benchmarks. Systematic ablations show that this advantage comes from the discrete token space, which allows early pruning and computational reuse, and our verifier analysis highlights trade-offs between speed and reasoning capability. These findings suggest that model architecture, not just scale, is critical for inference-time optimization in visual generation.

Figures (8)

• Figure 1: Guided search in autoregressive models provides a more efficient path to high-quality image generation than scaling diffusion models. (Left) ImageReward score vs. inference-time compute (NFEs). A 2B autoregressive model with beam search (green) surpasses a 12B FLUX.1-dev model ma2025inference using random search (orange), while requiring less computation. (Right) Examples showing how beam search corrects compositional errors in baseline generations. It successfully fixes object counts ("six keys"), incorrect spatial relationships ("giraffe on the right"), and color errors ("green rose and a blue tulip").
• Figure 2: Search Strategies for Autoregressive Image Generation. (a) Random search generates $n$ images independently and selects the one with the highest score. (b) Greedy token optimization operates sequentially; at each step, it generates $c$ candidate images and commits to the single token that produces the best result before continuing. (c) Beam search maintains $w$ parallel sequences. At each step, it explores $c$ options for each beam and keeps the top $w$ overall sequences, allowing it to explore more diverse paths than the greedy approach.
• Figure 3: Expected maximum verification scores as functions of budget size (log scale). All three verifiers exhibit logarithmic scaling.
• Figure 4: Qualitative comparison of verifiers on T2I-CompBench++. Each example shows the performance of the baseline and the beam search with ImageReward and LLaVA-OneVision. For attribute binding (e.g., Color), both verifiers perform well. For complex reasoning (e.g., Spatial), LLaVA-OneVision's capabilities are required to guide the model to a semantically correct image, correcting errors that ImageReward misses.
• Figure 5: Visual comparison of search strategies for text-to-image generation. Each row shows results for a different prompt, with columns representing: baseline generation (single sample with Infinity-2B), random search, greedy token optimization, and beam search. All images are generated using the Infinity-2B model with identical parameters. All search strategies have been guided by the ImageReward xu2023imagereward verifier. The budget is set to 390 verified images. All prompts are from Drawbench saharia2022photorealistic.