Inference-Time Scaling for Visual AutoRegressive modeling by Searching Representative Samples

TL;DR

The paper tackles the challenge of applying inference-time scaling to vector-quantized visual autoregressive (VAR) models, whose discrete latent spaces hinder continuous search. It introduces VAR-Scaling, a KDE-based method that maps discrete sampling spaces to quasi-continuous feature spaces and identifies representative samples via density estimation, using a density-adaptive Top-$k$/Random-$k$ strategy to balance quality and diversity. Across class-conditional and text-to-image benchmarks, VAR-Scaling yields meaningful gains in IS and maintains or improves FID/Geneval, outperforming baselines such as VAR and FlexVAR. The approach enables effective, scalable inference-time improvements for discrete visual generative models and opens avenues for extending density-guided sampling to other discrete-space generative tasks, with code available at the provided URL.

Abstract

While inference-time scaling has significantly enhanced generative quality in large language and diffusion models, its application to vector-quantized (VQ) visual autoregressive modeling (VAR) remains unexplored. We introduce VAR-Scaling, the first general framework for inference-time scaling in VAR, addressing the critical challenge of discrete latent spaces that prohibit continuous path search. We find that VAR scales exhibit two distinct pattern types: general patterns and specific patterns, where later-stage specific patterns conditionally optimize early-stage general patterns. To overcome the discrete latent space barrier in VQ models, we map sampling spaces to quasi-continuous feature spaces via kernel density estimation (KDE), where high-density samples approximate stable, high-quality solutions. This transformation enables effective navigation of sampling distributions. We propose a density-adaptive hybrid sampling strategy: Top-k sampling focuses on high-density regions to preserve quality near distribution modes, while Random-k sampling explores low-density areas to maintain diversity and prevent premature convergence. Consequently, VAR-Scaling optimizes sample fidelity at critical scales to enhance output quality. Experiments in class-conditional and text-to-image evaluations demonstrate significant improvements in inference process. The code is available at https://github.com/WD7ang/VAR-Scaling.

Figures (7)

• Figure 1: Diffusion models enable metric-guided continuous search from base to target distributions. In contrast, VQ's discrete nature prevents continuous path exploration. Our KDE mapping transforms probability-scaled sampling spaces into continuous trajectories, eliminating chaotic unguided searches evident.
• Figure 2: Scales 0-1 define general patterns like spatial structures and contours; scales 2-9 refine specific patterns with texture, edge, and local feature enhancements.
• Figure 3: For ImageNet generation, we report model performance using the standard evaluation metrics, FID $\downarrow$heusel2017gans and IS $\uparrow$salimans2016improved. Our experiments show that high-density samples are likely representative prototypes, while low-density samples may correspond to low quality outliers.
• Figure 4: Across successive expansion iterations (left → right), the representative samples of distribution exhibits progressive structural refinement.
• Figure 5: Experimental results demonstrate that high-density regions correspond to representative samples, whereas low-density samples are low-quality samples.