Odysseus Navigates the Sirens' Song: Dynamic Focus Decoding for Factual and Diverse Open-Ended Text Generation

TL;DR

This work tackles the persistent tension between factuality and diversity in open-ended text generation by introducing Dynamic Focus Decoding (DFD), a plug-and-play stochastic decoding framework. DFD computes a per-step knowledge-awareness signal from layer-wise distribution differences using the KL divergence, denoted $KA_t$, and transforms it into a dynamic temperature $T_t$ via Linear, Sigmoid-Scaled, or Exponential Decay mappings to steer sampling. It also provides a training variant, Focused Training (FT), with a loss that emphasizes knowledge-aware token selection. Across seven datasets and multiple decoding strategies, DFD yields consistent improvements in factuality and diversity with minimal computational overhead, and it robustly generalizes across model scales and architectures, while complementing fact-augmentation methods when desired.

Abstract

Large Language Models (LLMs) are increasingly required to generate text that is both factually accurate and diverse across various open-ended applications. However, current stochastic decoding methods struggle to balance such objectives. We introduce Dynamic Focus Decoding (DFD), a novel plug-and-play stochastic approach that resolves this trade-off without requiring additional data, knowledge, or models. DFD adaptively adjusts the decoding focus based on distributional differences across layers, leveraging the modular and hierarchical nature of factual knowledge within LLMs. This dynamic adjustment improves factuality in knowledge-intensive decoding steps and promotes diversity in less knowledge-reliant steps. DFD can be easily integrated with existing decoding methods, enhancing both factuality and diversity with minimal computational overhead. Extensive experiments across seven datasets demonstrate that DFD significantly improves performance, providing a scalable and efficient solution for open-ended text generation.

Figures (4)

• Figure 1: Adaptive focus adjustment in stochastic decoding to balance factuality and diversity.
• Figure 2: Distributional differences across layers during decoding for knowledge-aware (e.g., Isaac Newton) and non-knowledge-aware (e.g., "sir," "was") steps. The final row displays the predicted tokens at each decoding step, with the intensity of knowledge awareness represented by the color gradient. The other row names correspond to the indices of the internal layers utilized.
• Figure 3: General chatbot performance comparison. Left: Counts of wins, ties, and losses. Right: Average scores of our method and the baseline.
• Figure 4: Robustness of DFD across different decoding settings for four stochastic decoding algorithms. The dark portion of each bar indicates the baseline performance, while the light portion above shows the improvement achieved by DFD, with numeric values annotated.