StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Models

TL;DR

This work proposes StepVAR, a training-free token pruning framework that accelerates VAR inference by jointly considering structural and textural importance and outperforms existing acceleration approaches, validating its effectiveness and general applicability across diverse VAR architectures.

Abstract

Visual AutoRegressive (VAR) models based on next-scale prediction enable efficient hierarchical generation, yet the inference cost grows quadratically at high resolutions. We observe that the computationally intensive later scales predominantly refine high-frequency textures and exhibit substantial spatial redundancy, in contrast to earlier scales that determine the global structural layout. Existing pruning methods primarily focus on high-frequency detection for token selection, often overlooking structural coherence and consequently degrading global semantics. To address this limitation, we propose StepVAR, a training-free token pruning framework that accelerates VAR inference by jointly considering structural and textural importance. Specifically, we employ a lightweight high-pass filter to capture local texture details, while leveraging Principal Component Analysis (PCA) to preserve global structural information. This dual-criterion design enables the model to retain tokens critical for both fine-grained fidelity and overall composition. To maintain valid next-scale prediction under sparse tokens, we further introduce a nearest neighbor feature propagation strategy to reconstruct dense feature maps from pruned representations. Extensive experiments on state-of-the-art text-to-image and text-to-video VAR models demonstrate that StepVAR achieves substantial inference speedups while maintaining generation quality. Quantitative and qualitative evaluations consistently show that our method outperforms existing acceleration approaches, validating its effectiveness and general applicability across diverse VAR architectures.

Figures (7)

• Figure 1: Inference Characteristics of Visual Autoregressive Models. (a) The computational bottleneck is heavily skewed towards the final resolution scales, which account for the majority of latency. (b) Sensitivity analysis reveals that early scales are highly sensitive to structural perturbations, whereas later scales exhibit high robustness and error tolerance. (c) Visualizing the residual difference between stages demonstrates that the final scales primarily serve to refine high-frequency textural details.
• Figure 2: Overview of the StepVAR Framework. (a) Overall Pipeline: Input tokens are pruned prior to entering the transformer blocks. In the Recover stage, the dense feature map required for the next scale is reconstructed using Nearest Neighbor Feature Propagation. (b) Token Pruning: Our Structure-Texture Guided strategy calculates a joint importance score by weighting a Structural Score ($\mathbf{S}_{\text{str}}$), derived from PCA analysis, and a Textural Score ($\mathbf{S}_{\text{txt}}$), derived from spatial high-frequency filter. This ensures the retention of tokens critical for both global layout and local details.
• Figure 3: Correlation of first principle component and transformer block residual.
• Figure 4: Visual Comparison between original model and our StepVAR generated images.
• Figure 5: Visual comparison of different recovery strategies.