Fast SceneScript: Accurate and Efficient Structured Language Model via Multi-Token Prediction

TL;DR

The paper tackles the slow decoding speed of language-based 3D scene layout estimation by introducing Fast SceneScript, which employs multi-token prediction to reduce autoregressive steps. To maintain accuracy, it pairs MTP with self-speculative decoding and a confidence-guided decoding mechanism, plus a parameter-efficient head-sharing design. Across ASE and Structured3D, Fast SceneScript yields up to ~5× speedups with minimal parameter overhead and improved or maintained F1 scores compared to previous SceneScript variants. The approach demonstrates that predicting multiple tokens in structured visual language is viable for real-time 3D perception tasks and can generalize to other perception problems.

Abstract

Recent perception-generalist approaches based on language models have achieved state-of-the-art results across diverse tasks, including 3D scene layout estimation, via unified architecture and interface. However, these approaches rely on autoregressive next-token prediction, which is inherently slow. In this work, we introduce Fast SceneScript, a novel structured language model for accurate and efficient 3D scene layout estimation. Our method employs multi-token prediction (MTP) to reduce the number of autoregressive iterations and significantly accelerate inference. While MTP improves speed, unreliable token predictions can significantly reduce accuracy. To filter out unreliable tokens, we adapt self-speculative decoding (SSD) for structured language models and introduce confidence-guided decoding (CGD) with an improved scoring mechanism for token reliability. Furthermore, we design a parameter-efficient mechanism that reduces the parameter overhead of MTP. Extensive experiments on the ASE and Structured3D benchmarks demonstrate that Fast SceneScript can generate up to 9 tokens per decoder inference step without compromising accuracy, while adding only $\sim7.5\%$ additional parameters.

Figures (8)

• Figure 1: Comparison between SceneScript avetisyan2024SceneScript, SceneScript avetisyan2024SceneScript + MTP gloeckle2024better, and Fast SceneScript. Circle area is proportional to the number of decoder parameters. MTP-$n$ refers to SceneScript avetisyan2024SceneScript with $n$ token heads predicting $n$ future tokens. Fast SceneScript exhibits markedly higher computational efficiency than SceneScript avetisyan2024SceneScript, achieving a $5.09\times$ speedup. Additionly, our method delivers a $12.04\%$ improvement in F1-Score while utilizing $43\%$ fewer parameters compared to SceneScript avetisyan2024SceneScript + MTP gloeckle2024better.
• Figure 2: Comparison between our proposed Fast SceneScript and SceneScript avetisyan2024SceneScript. Gray vertical line in (b) and (c) indicates the separation between different forward passes. To generate the same sequence, SceneScript avetisyan2024SceneScript requires $21$ decoder iterations, whereas our method completes this in just $3$ iterations (producing $8$, $7$, and $6$ tokens per iteration), delivering significantly higher efficiency.
• Figure 3: Framework of our proposed Fast SceneScript. The 3D point cloud is initially processed by the 3D encoder. Subsequently, the extracted 3D features, together with a sequence of preceding tokens, i.e., $\{t_{1}, t_{2}, \dots, t_{k}\}$, are fed into the language decoder to generate the next $n$ tokens $\{t_{k+1}, t_{k+2}, \dots, t_{k+n}\}$ and $(n-1)$ confidences $\{c_{k+2}, ..., c_{k+n}\}$. Projection block, token head, and confidence head are shared across all $n$ tokens. To improve prediction reliability, token filtering strategy removes unreliable tokens.
• Figure 4: Token filtering strategies. This figure illustrates an example where the first two generated tokens are accepted. (a) Self-Speculative Decoding DBLP:conf/acl/Zhang00S0CM24stern2018blockwise: The language decoder drafts next $n$ tokens $\{t_{k+1}, t_{k+2}, ..., t_{k+n}\}$ conditioned on preceding tokens $t_{\leq k}$. Then, $t_{\leq k+n-1}$ are fed to the language decoder again for next-token prediction, generating $\{\widetilde{t}_{k+i}\}_{i=1}^n$ using the first token head. Finally, consistency checking is applied between $\widetilde{t}_{k+i}$ and $t_{k+i}$, after which only tokens before the first unreliable token are retained. (b) Confidence-Guided Decoding: The language decoder predicts both tokens and their confidences. Decoding terminates upon encountering the first token deemed unreliable.
• Figure 5: Qualitative comparisons on ASE test set avetisyan2024SceneScript. The number of MTP heads $n$ is set to 8. Compared to SceneScript avetisyan2024SceneScript + MTP gloeckle2024better, our Fast SceneScript can predict more complete and accurate scene layouts.