LANTERN++: Enhancing Relaxed Speculative Decoding with Static Tree Drafting for Visual Auto-regressive Models

TL;DR

This work targets the slowdown of visual auto-regressive decoding caused by token selection ambiguity. It introduces LANTERN++, a framework that leverages static tree drafting and a multiplicative relaxation bound to decouple draft selection from low-confidence predictions, enabling deeper draft sequences. Empirical results across multiple visual AR models and the MS-COCO dataset show substantial speedups (up to $\times 2.56$ latency reduction and $\times 3.63$ step compression) with only modest degradation in image quality, demonstrating practical gains for real-time visual generation. The approach preserves distributional fidelity while unlocking more aggressive speculative decoding, offering a scalable path to faster visual AR generation.

Abstract

Speculative decoding has been widely used to accelerate auto-regressive (AR) text generation. However, its effectiveness for visual AR models remains limited due to token selection ambiguity, where multiple tokens share similarly low probabilities and thus reduce acceptance rates. Recently, relaxed speculative decoding with dynamic tree drafting was proposed to mitigate this ambiguity, demonstrating promising results in accelerating visual AR models. However, we observe that token selection ambiguity still negatively affects dynamic tree drafting, resulting in shallow draft trees and limited acceleration. To overcome this issue, we introduce LANTERN++, a refined framework that integrates static tree drafting with a tailored relaxed acceptance condition, allowing drafts to be selected independently of low-confidence predictions. This enables the acceptance of deeper sequences, improving decoding efficiency while preserving image quality. Extensive experiments on state-of-the-art visual AR models demonstrate that LANTERN++ significantly accelerates inference, achieving up to $\mathbf{\times 2.56}$ speedup over standard AR decoding while maintaining high image quality. The code is publicly available at https://github.com/jadohu/LANTERN.

Figures (5)

• Figure 1: Images generated by LANTERN++ on Lumina-mGPT liu2024luminamgpt exhibit remarkable acceleration in decoding steps while maintaining high generation quality. The top row shows images generated by standard auto-regressive decoding, while the bottom row displays images generated with acceleration through LANTERN++. The step compression ratio is presented in white at the bottom right of each image.
• Figure 2: Images generated by LANTERN++ on Lumina-mGPT with average step compression ratio $\times 3.63$. The top row shows images generated by standard auto-regressive decoding, and the bottom row displays images generated with acceleration through LANTERN++.
• Figure 3: Comparison of static tree structures. Top: The original static tree used in EAGLE-1. Bottom: The extended static tree used for both EAGLE-1 and LANTERN++ in our experiments, designed to match the scale of dynamic tree drafting while maintaining EAGLE-1's structural principles.
• Figure 4: Example of a shallow draft tree produced by dynamic tree drafting. Due to low drafter confidence, deeper expansions are rarely selected, resulting in a tree that is wide but lacks depth, which hinders step compression.
• Figure 5: Comparison of images generated using additive relaxation ($\delta$) and multiplicative relaxation ($\lambda$). The left column shows images generated without relaxation. The middle and right columns show images with increasing levels of $\delta$ and $\lambda$, respectively. While both approaches increase acceleration, additive relaxation leads to more noticeable artifacts, whereas multiplicative relaxation maintains stable image quality.