Adaptive Visual Autoregressive Acceleration via Dual-Linkage Entropy Analysis

TL;DR

This work tackles the high computational cost of Visual AutoRegressive (VAR) generation stemming from massive token counts. It introduces NOVA, a training-free framework that uses entropy-guided dual-linkage token reduction to adaptively prune tokens across both scales and Transformer layers, activating at an entropy inflection point and reusing residual caches to preserve structure. The approach combines scale-linkage and layer-linkage ratio functions with residual cache reuse to achieve substantial speedups (e.g., up to 2.89× on Infinity-2B) while maintaining or even improving perceptual and semantic quality across multiple backbones and benchmarks. Ablation studies show entropy-based pruning outperforms attention-based or MSE-based criteria, and the dual-linkage design consistently outperforms single-linkage variants, highlighting the method's practical impact for faster high-resolution visual autoregressive generation without retraining.

Abstract

Visual AutoRegressive modeling (VAR) suffers from substantial computational cost due to the massive token count involved. Failing to account for the continuous evolution of modeling dynamics, existing VAR token reduction methods face three key limitations: heuristic stage partition, non-adaptive schedules, and limited acceleration scope, thereby leaving significant acceleration potential untapped. Since entropy variation intrinsically reflects the transition of predictive uncertainty, it offers a principled measure to capture modeling dynamics evolution. Therefore, we propose NOVA, a training-free token reduction acceleration framework for VAR models via entropy analysis. NOVA adaptively determines the acceleration activation scale during inference by online identifying the inflection point of scale entropy growth. Through scale-linkage and layer-linkage ratio adjustment, NOVA dynamically computes distinct token reduction ratios for each scale and layer, pruning low-entropy tokens while reusing the cache derived from the residuals at the prior scale to accelerate inference and maintain generation quality. Extensive experiments and analyses validate NOVA as a simple yet effective training-free acceleration framework.

Figures (8)

• Figure 1: NOVA generates a 2K image in 3.2 seconds on a single NVIDIA RTX 3090 GPU without running out of GPU memory.
• Figure 2: (a) Peak GPU memory consumption across different scales during 1024×1024 image generation in VAR. As the scale increases, the token sequence length grows rapidly, leading to quadratic growth in attention computation complexity. (b) Theoretical GFLOPs required by next-token and next-scale prediction models to generate images of various resolutions under uniform settings. Compared to the raster-scan next-token prediction, VAR's coarse-to-fine next-scale prediction additionally requires predicting feature maps at multiple previous scales before generating the final scale. The cumulative computation at these previous transition scales introduces extra computational cost to VAR.
• Figure 3: Entropy heatmaps of visualized images and token maps.
• Figure 4: (a) Frequency spectrum of various instances at different scales during VAR generation. (b) Various entropy values across different scales during the inference process of Infinity-2B: We selected 600 prompts from GenEval and rewrote them. For each rewritten prompt, we generated four images using different random seeds. (c) Entropy heatmaps of token maps across different Transformer layers. Cross-layer spatial pattern variations: Shallow layers exhibit high-entropy regions resembling local noise or scattered dots, certain middle layers show high-entropy regions with clear subject contours, while deeper layers display regular periodic grid-like patterns in high-entropy regions. Cross-layer overall entropy variations: For example, token map at the 19-th layer has relatively low overall entropy.
• Figure 5: NOVA framework at both scale-level and layer-level.