ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training
Feijiang Han, Xiaodong Yu, Jianheng Tang, Delip Rao, Weihua Du, Lyle Ungar
TL;DR
This work addresses improving frozen LLMs at inference without training by tuning the initial token's attention. It formalizes how scaling the initial token's attention with a factor $\gamma$ reshapes downstream attention via $a_0' = \frac{\gamma a_0}{D}$ and $a_i' = \frac{a_i}{D}$ with $D = (\gamma-1)a_0 + 1$, showing the effect grows with $a_0$ and can sharpen or smooth attention distributions. Building on this, the authors introduce ZeroTuning, a training-free, kernel-agnostic method that applies head-specific attention adjustments to the initial token, with supervised and unsupervised variants (entropy minimization). Across four LLMs and 15 benchmarks, ZeroTuning yields substantial gains on text classification, domain-specific questions, and multi-round conversations, while remaining effective under long contexts and quantization. The method is robust, practical, and complementary to supervised fine-tuning, highlighting the initial token as a powerful, deployment-friendly control point for model adaptation.$
Abstract
Token-level attention tuning, a class of training-free methods including Post-hoc Attention Steering (PASTA) and Attention Calibration (ACT), has emerged as a promising approach for improving frozen LLMs via interpretable interventions. However, these methods rely on auxiliary heuristics to identify important task-specific tokens, which can introduce bias and limit applicability when token importance is ambiguous or when optimized kernels make attention maps inaccessible. We propose a simpler alternative: intervening only on the initial token (e.g., BOS in LLaMA). We theoretically show that adding lightweight biases to this token's attention logits systematically shifts and reshapes downstream attention patterns - an effect amplified by its natural role as an attention sink. Empirically, we find that this tuning can improve LLM performance and better elicit pretrained knowledge, with stronger effects in early layers and distinct scaling preferences across attention heads. Building on these findings, we introduce ZeroTuning, a training-free method that improves LLM performance by applying head-specific attention adjustments to the initial token, requiring no parameter updates. We present two variants: a supervised mode that calibrates on validation examples, and an unsupervised mode that directly minimizes output entropy. ZeroTuning requires no KV-cache or decoding changes and is kernel-agnostic (works with SDPA and FlashAttention). It requires only four lines of modification to the standard LlamaAttention code, achieves gains across 15 datasets, and outperforms prior, more complex methods. For example, on Llama-3.1-8B, it yields relative improvements of 19.9% on classification, 4.5% on question answering, and 2.1% on dialogue. ZeroTuning also works out of the box with quantized inference and maintains its improvements as context length increases.