Inference-Time Language Model Alignment via Integrated Value Guidance

TL;DR

The paper tackles the resource-intensive problem of aligning large language models with human preferences by proposing Integrated Value Guidance (IVG), an inference-time approach that uses a small implicit value function at the token level and a small explicit value function at the chunk level to steer decoding without fine-tuning. IVG combines token-wise sampling and chunk-level beam search, trained via Direct Preference Optimization and FUDGE, respectively, to outperform baselines across sentiment, summarization, and instruction-following tasks. Key findings show that explicit chunk-level guidance excels for larger models while implicit token-level guidance benefits finer-grained control, and their integration yields the strongest overall alignment. The work demonstrates practical, scalable, and flexible alignment possible entirely during decoding, reducing retraining needs and enabling per-task or per-user preference customization.

Abstract

Large language models are typically fine-tuned to align with human preferences, but tuning large models is computationally intensive and complex. In this work, we introduce $\textit{Integrated Value Guidance}$ (IVG), a method that uses implicit and explicit value functions to guide language model decoding at token and chunk-level respectively, efficiently aligning large language models purely at inference time. This approach circumvents the complexities of direct fine-tuning and outperforms traditional methods. Empirically, we demonstrate the versatility of IVG across various tasks. In controlled sentiment generation and summarization tasks, our method significantly improves the alignment of large models using inference-time guidance from $\texttt{gpt2}$-based value functions. Moreover, in a more challenging instruction-following benchmark AlpacaEval 2.0, we show that both specifically tuned and off-the-shelf value functions greatly improve the length-controlled win rates of large models against $\texttt{gpt-4-turbo}$ (e.g., $19.51\% \rightarrow 26.51\%$ for $\texttt{Mistral-7B-Instruct-v0.2}$ and $25.58\% \rightarrow 33.75\%$ for $\texttt{Mixtral-8x7B-Instruct-v0.1}$ with Tulu guidance).

Figures (6)

• Figure 1: Illustration of Integrated Value Guidance (IVG) with parameters $W,K,L=2,2,30$. Weak-to-Strong Search zhou2024weak denotes the results with the same parameters. BoNe denotes Best-of-N Sampling ($N=4$) with explicit values.
• Figure 2: Illustration of Integrated Value Guidance (IVG) with beam width $W=2$, successors per state $K=2$, and chunk length $L=5$.
• Figure 3: The gold reward achieved for different large pre-trained models under the gpt2 guidance. We show the mean reward across three random seeds. BoNi and BoNe denote BoN ($N=16$) with implicit and explicit rewards, respectively; EFT ($\beta^*$) denotes the best EFT results among $\beta \in \{0.25, 0.5, 1, 2\}$; CBS denotes the results with $W,K,L=4,4,5$ and implicit rewards; IVG denotes the best results with $W,K,L=4,4,5$ among $\beta \in \{0.25, 0.5, 1, 2\}$.
• Figure 4: The performance of different combinations of implicit and explicit value functions for token-wise sampling and chunk-level beam search in controlled-sentiment generation and summarization. "implicit" and "explicit" denotes applying implicit and explicit value functions. "none" denotes the base model without any guidance. The number denotes the gold reward for the corresponding combination.
• Figure 5: The length-controlled win rates (LC Win Rates) against gpt-4-turbo for various instruction-tuned models under TuluGuidance and UltraGuidance. BoNe denote BoN ($N=16$) with explicit rewards, respectively; EFTi ($\beta^*$) denotes the results with $\beta_i=1.0$ for Tulu guidance and $\beta_i=1.5$ for Ultra guidance which are the best parameters we evaluated on Llama-2-7b-chat-hf. CBSi and CBSe denote the results with implicit and explicit value functions with $W,K,L=2,2,30$. IVG denotes the best results with $W,K,L=2,2,30$ with $\beta=\beta_i$. BoNi and EFTe are not shown due to their weak performance. More results are shown in appendix \ref{['app:sec:extened-exp-results']}.