Fast Inference of Visual Autoregressive Model with Adjacency-Adaptive Dynamical Draft Trees

TL;DR

This work tackles the slow inference of visual autoregressive models by adapting speculative decoding to the spatially heterogeneous nature of images. It introduces ADT-Tree, an adjacency-adaptive dynamical draft-tree framework that initializes draft trees from adjacent tokens and then dynamically adjusts depth and width based on observed acceptance rates, balancing computation and accuracy. Two-phase design—Adjacent Initialization and Bisectional Dynamic Adaptation—enables deeper trees in simple regions and wider trees in complex ones, achieving up to 3.13× speedups (MSCOCO2017) and 3.05× (PartiPrompts) while preserving image quality, and it can integrate with relaxed sampling approaches like LANTERN. The approach is evaluated on large text-conditioned image-generation benchmarks, showing robust acceleration with minimal quality loss and offering a practical, plug-and-play acceleration for visual AR models.

Abstract

Autoregressive (AR) image models achieve diffusion-level quality but suffer from sequential inference, requiring approximately 2,000 steps for a 576x576 image. Speculative decoding with draft trees accelerates LLMs yet underperforms on visual AR models due to spatially varying token prediction difficulty. We identify a key obstacle in applying speculative decoding to visual AR models: inconsistent acceptance rates across draft trees due to varying prediction difficulties in different image regions. We propose Adjacency-Adaptive Dynamical Draft Trees (ADT-Tree), an adjacency-adaptive dynamic draft tree that dynamically adjusts draft tree depth and width by leveraging adjacent token states and prior acceptance rates. ADT-Tree initializes via horizontal adjacency, then refines depth/width via bisectional adaptation, yielding deeper trees in simple regions and wider trees in complex ones. The empirical evaluations on MS-COCO 2017 and PartiPrompts demonstrate that ADT-Tree achieves speedups of 3.13xand 3.05x, respectively. Moreover, it integrates seamlessly with relaxed sampling methods such as LANTERN, enabling further acceleration. Code is available at https://github.com/Haodong-Lei-Ray/ADT-Tree.

Figures (9)

• Figure 1: The comparison of the training-set accuracy of draft models for the text-to-text and the text-to-image. Vicuna-7B Vicuna, LLaMA2-Chat-7B and LLaMA2-Chat-13B llamachat2 are text-to-text generation models. Llamagen-XL, Anole-7B Anole, and Lumina-mGpt-7B are text-to-image generation models. Both draft models are built upon the EAGLE draft model architecture EAGLE-1.
• Figure 2: The draft model faces two situations in different image regions. The image token depth matrix tracks the depth of the draft tree at which each image token resides. In this matrix, brighter areas signify deeper locations of the image tokens within the draft tree. For complex regions, the acceptance length is lower than the height of the draft tree, making unused layers wasteful and reducing the acceleration rate. A shallow draft tree is appropriate. For simple regions, the potential acceptance length exceeds the draft tree height, so building a deeper tree can increase the acceptance length and boost the acceleration rate.
• Figure 3: (a) Left: Frequency of acceptance lengths during speculative decoding with $\mathcal{T}_{\text{draft}}$ ($\hat{d} = 5$, $\hat{k} = 10$) over 100 image generations using Anole at $T = 0$ and $T = 1$. (b) Right: Frequency of the top-$k$ positions of accepted draft tokens, where 'Top-$k$ Location' denotes the minimum $k_d$ required for $\mathcal{R}$ to include the correct token in the draft phase for acceptance by $\mathcal{L}$.
• Figure 4: (a) The original image generated from 32×32 image tokens.(b) The top-1 probability values of the generation distribution for each of the 32×32 image tokens. The model is Anole, with the prompt "a zebra".
• Figure 5: (a) The probability distribution for simpler texture regions. It corresponds to the position (8,1) in Figure \ref{['fig:intro2.2']}(a). (d) The probability distribution for complex texture regions. It corresponds to the position (9,20) in Figure \ref{['fig:intro2.2']}(b). It can be observed that the probability distribution for simpler texture regions is more uniform, with the maximum value not exceeding 0.01, whereas the probability distribution for complex texture regions is significantly more peaked, with the maximum value surpassing 0.14.