Temporal Guidance for Large Language Models

TL;DR

Temporal Guidance (TeGu) introduces a memory-efficient decoding strategy that uses temporally offset Multi-Token Prediction heads as internal amateurs to contrast against the expert Next Token Prediction. A lightweight Conditional MTP Projector (cMTPP) enables mainstream LLMs to exploit MTP-like signals without external models, trained with a CE+KD objective to align amateur and expert distributions. Across mathematics, coding, and instruction-following benchmarks, TeGu consistently outperforms Greedy decoding, standard Contrastive Decoding, and DoLa, with strong robustness on smaller models and favorable efficiency metrics. The approach emphasizes improved coherence and reasoning while maintaining low memory and latency overhead, though it shows neutral impact on factual benchmarks like TruthfulQA and requires training overhead for non-MTP architectures.

Abstract

Contrastive Decoding (CD) enhances the generation quality of large language models (LLMs) but incurs significant additional computational overhead due to the need for an auxiliary model. Existing internal self-contrastive decoding methods, such as Decoding by Contrasting Layers (DoLa), focus on discrepancies across different layers, which are notably unstable on small-scale models. In this work, based on the observation that LLMs exhibit local preferences, we propose a novel contrastive guidance strategy along the temporal dimension, namely Temporal Guidance (TeGu). Our method ingeniously leverages Multi-Token Prediction (MTP) to construct weaker amateur predictions for model self-contrast. To standardize the implementation of this mechanism, we further introduce a lightweight Conditional MTP Projector (cMTPP), which avoids maintaining multiple independent networks as required by other MTP modules. Across various model series and benchmarks, TeGu achieves significant performance improvements while maintaining low additional memory consumption and computational overhead.

Figures (4)

• Figure 1: Overview of Temporal Guidance (TeGu). Left: TeGu leverages the contrast between the MTP outputs from previous steps (Amateur) and the NTP from the current step (Expert). It performs logits update using the formula $(1+\alpha)\log P_{\text{exp}} - \alpha \log P_{\text{amt}}$. Subsequently, the next token is obtained based on the updated logits. Right: The Conditional MTP Projector employs AdaLN to inject the future time step offset (Step ID) into the hidden states, which are then processed through a SwiGLU FFN. The frozen base language model head is reused to obtain the MTP outputs.
• Figure 2: Ablation study on training objectives. We evaluate TeGu on Llama-3.2-3B across four benchmarks. The results compare the Baseline against the MTP projector trained with only Cross-Entropy loss (CE Loss) and our proposed method combining CE and KL divergence (CE Loss + KL Loss).
• Figure 3: The distribution entropy density histograms of NTP and MTP heads for the Qwen3-8B model on the MATH500.
• Figure 4: Ablation study on the hyperparameter $\alpha$. The figure reports the Mean Accuracy of Qwen3-1.7B and Qwen3-8B models under different $\alpha$ values. Error bars indicate the standard error.