Integrative Decoding: Improve Factuality via Implicit Self-consistency

TL;DR

This work tackles factuality in large language models by addressing the limitations of prior self-consistency approaches for open-ended generation. It introduces Integrative Decoding (ID), an inference-time method that constructs multiple inputs by prepending sampled responses to the prompt and decodes in parallel, aggregating logits to implicitly encourage self-consistency. Across six diverse LLMs and three open-ended benchmarks (TruthfulQA, Biographies, LongFact), ID yields substantial factuality gains and its benefits scale with the number of sampled responses, while maintaining reasonable efficiency. The results suggest that ID offers a practical path toward inference-time scaling of factual accuracy in real-world generation tasks, with robust performance across model scales and sampling strategies, and it points to promising future work combining ID with speculative decoding and more precise decoding strategies.

Abstract

Self-consistency-based approaches, which involve repeatedly sampling multiple outputs and selecting the most consistent one as the final response, prove to be remarkably effective in improving the factual accuracy of large language models. Nonetheless, existing methods usually have strict constraints on the task format, largely limiting their applicability. In this paper, we present Integrative Decoding (ID), to unlock the potential of self-consistency in open-ended generation tasks. ID operates by constructing a set of inputs, each prepended with a previously sampled response, and then processes them concurrently, with the next token being selected by aggregating of all their corresponding predictions at each decoding step. In essence, this simple approach implicitly incorporates self-consistency in the decoding objective. Extensive evaluation shows that ID consistently enhances factuality over a wide range of language models, with substantial improvements on the TruthfulQA (+11.2%), Biographies (+15.4%) and LongFact (+8.5%) benchmarks. The performance gains amplify progressively as the number of sampled responses increases, indicating the potential of ID to scale up with repeated sampling.

Figures (9)

• Figure 1: With no need of retrieving external knowledge and additional training, integrative decoding consistently improves the factuality performance over six types of large language models, with substantial improvements on the TruthfulQA, Biographies, and LongFact datasets.
• Figure 2: The workflow of integrative decoding: (1) sample multiple responses from the LLM; (2) form a set of new inputs by concatenating a sampled response and the original prompt; they are concurrently processed for decoding, with the next token being selected by integrating their predicted logits at each inference step. This strategy essentially incorporates the overall consistency with all sampled responses in its decoding objective (see Section \ref{['sec:method']}).
• Figure 3: The performance of ID on different model scales from the Qwen-2.5 series. Additional results for the LLaMA and Mistral series are shown in Figure \ref{['fig:model_scale']}.
• Figure 4: The performance of different approaches on the Biographies dataset over six LLMs, when the number of sampled responses is 1, 4, 8, 12, and 16, respectively.
• Figure 5: The performance of ID, with sampled responses obtained via different sampling strategies (temperature sampling with $T\in\{0.3, 0.5, 0.7\}$ and nucleus sampling with $p\in\{0.9, 0.95\}$). The best results and the runner-ups are highlighted in blue and green.