Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Training

TL;DR

The paper tackles the latency bottleneck of autoregressive decoding in large language models by introducing Make Some Noise (MSN), a denoising-based replacement for supervised fine-tuning that enables parallel draft-token generation. It pairs MSN with TR-Jacobi decoding, a tree- and retrieval-augmented strategy that further accelerates inference without adding extra model structures or post-training requirements. Empirical results show 2.3–2.7x speedups in general and code domains, with competitive performance on Spec-Bench compared to state-of-the-art methods that rely on additional architectures. This approach offers a lightweight, deployment-friendly path to faster inference on LLMs while preserving task capabilities, and it highlights the practical value of treating parallel decoding as a robust denoising problem during training.

Abstract

Existing speculative decoding methods typically require additional model structure and training processes to assist the model for draft token generation. This makes the migration of acceleration methods to the new model more costly and more demanding on device memory. To address this problem, we propose the Make Some Noise (MSN) training framework as a replacement for the supervised fine-tuning stage of the large language model. The training method simply introduces some noise at the input for the model to learn the denoising task. It significantly enhances the parallel decoding capability of the model without affecting the original task capability. In addition, we propose a tree-based retrieval-augmented Jacobi (TR-Jacobi) decoding strategy to further improve the inference speed of MSN models. Experiments in both the general and code domains have shown that MSN can improve inference speed by 2.3-2.7x times without compromising model performance. The MSN model also achieves comparable acceleration ratios to the SOTA model with additional model structure on Spec-Bench.

Figures (7)

• Figure 1: An illustration of the differences between the proposed MSN framework and existing model-based speculative decoding methods. The book icon represents task-specific capabilities and the rocket icon represents parallel decoding capabilities.
• Figure 2: Illustrations of the Make Some Noisy training framework and Jacobi decoding strategy. The training phase in the figure uses a noise segment of length 2, and the inference phase is shown as an example when the length of the noise segment is set to 3.
• Figure 3: The main flowchart of TR-Jacobi decoding. It should be noted that candidate generation and tree verification are performed in the same step. For clarity, we choose candidate generation at moment T and tree verification at moment T+1 for analysis in the figure.
• Figure 4: The effect of the inference noise segments length on acceleration with Jacobi decoding. '#MAT': Mean Accepted Token.
• Figure 5: Results of ablation experiments on the retrieval part of TR-Jacobi decoding.