Evolutionary Guided Decoding: Iterative Value Refinement for LLMs

TL;DR

This work tackles the distributional gap in value-function training that limits value-guided decoding for LLMs. It introduces Iterative Value Refinement (IVR), which combines Value Exploration and Iterative Self-Refinement to progressively improve the value function without retraining the base policy. Through experiments on summarization, multi-turn dialogue, and instruction following, IVR consistently outperforms decoding-time baselines while reducing computational overhead. The approach also demonstrates value-function transferability across model sizes and enhances safety against jailbreaking, highlighting practical benefits for flexible, cost-efficient model alignment.

Abstract

While guided decoding, especially value-guided methods, has emerged as a cost-effective alternative for controlling language model outputs without re-training models, its effectiveness is limited by the accuracy of the value function. We identify that this inaccuracy stems from a core distributional gap: existing methods train static value functions on trajectories sampled exclusively from the base policy, which inherently confines their training to a narrow and suboptimal view of the potential output space. We propose Iterative Value Refinement, a novel framework designed to bridge this gap. It employs Value Exploration to provide a more comprehensive and robust training signal, complemented by Iterative Self-Refinement, which uses the improved value function from one iteration to guide the generation of higher-quality data for the next. Extensive experiments on text summarization, multi-turn dialogue, and instruction following demonstrate the effectiveness of our framework in aligning language models. Our approach not only achieves alignment but also significantly reduces computational costs by leveraging principled value function optimization for efficient and effective control.

Figures (6)

• Figure 1: (a) Conventional methods suffer from a distributional gap, limiting the guided policy from reaching optimal rewards. (b) Our Iterative Value Refinement framework bridges this gap through Value Exploration and Iterative Self-Refinement.
• Figure 2: Reward vs. KL divergence for different methods on (a) summarization and (b) multi-turn dialogue.
• Figure 3: Comparison of different value functions using value-guided blockwise beam search on summarization (left), multi-turn dialogue (middle) and instruction following (right). IVR$^*$ denotes IVR without Iterative Self-Refinement.
• Figure 4: Ablation studies on (a) the number of sampled trajectories and (b) the number of training iterations in multi-turn dialogue using blockwise beam search. ST denotes sampled trajectories, TI denotes training iterations.
• Figure 5: Comparison of reward vs. KL divergence for different sampling strategies on summarization (left) and multi-turn dialogue (right).