Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Consistent Decoding for More Factual Open Responses

Christopher Malon, Xiaodan Zhu

TL;DR

The paper addresses factuality errors in open-ended generation by extending self-consistency to full outputs through Sample & Select, a per-sentence decoding strategy that selects sentences from multiple samples based on a token-overlap consensus and conditions later sentences on earlier choices. The approach uses nucleus sampling to generate multiple candidate sentences, scores them with a lightweight overlap-based metric, and iteratively builds the final summary with sentence-level conditioning. Empirical results on CNN/DM and XSum (FRANK subset) show large relative gains in automatic factuality metrics (SummaCZS and QAFE) while maintaining competitive ROUGE scores, with human evaluators also rating Sample & Select highest in factual accuracy. The method is model- and task-agnostic, computationally efficient, and comes with released code and evaluation data, offering a practical path to reduce open-ended hallucinations in generation systems.

Abstract

Self-consistency has emerged as a powerful method for improving the accuracy of short answers generated by large language models. As previously defined, it only concerns the accuracy of a final answer parsed from generated text. In this work, we extend the idea to open response generation, by integrating voting into the decoding method. Each output sentence is selected from among multiple samples, conditioning on the previous selections, based on a simple token overlap score. We compare this "Sample & Select" method to greedy decoding, beam search, nucleus sampling, and the recently introduced hallucination avoiding decoders of DoLA, P-CRR, and S-CRR. We show that Sample & Select improves factuality by a 30% relative margin against these decoders in NLI-based evaluation on the subsets of CNN/DM and XSum used in the FRANK benchmark, while maintaining comparable ROUGE-1 F1 scores against reference summaries. We collect human verifications of the generated summaries, confirming the factual superiority of our method.

Self-Consistent Decoding for More Factual Open Responses

TL;DR

The paper addresses factuality errors in open-ended generation by extending self-consistency to full outputs through Sample & Select, a per-sentence decoding strategy that selects sentences from multiple samples based on a token-overlap consensus and conditions later sentences on earlier choices. The approach uses nucleus sampling to generate multiple candidate sentences, scores them with a lightweight overlap-based metric, and iteratively builds the final summary with sentence-level conditioning. Empirical results on CNN/DM and XSum (FRANK subset) show large relative gains in automatic factuality metrics (SummaCZS and QAFE) while maintaining competitive ROUGE scores, with human evaluators also rating Sample & Select highest in factual accuracy. The method is model- and task-agnostic, computationally efficient, and comes with released code and evaluation data, offering a practical path to reduce open-ended hallucinations in generation systems.

Abstract

Self-consistency has emerged as a powerful method for improving the accuracy of short answers generated by large language models. As previously defined, it only concerns the accuracy of a final answer parsed from generated text. In this work, we extend the idea to open response generation, by integrating voting into the decoding method. Each output sentence is selected from among multiple samples, conditioning on the previous selections, based on a simple token overlap score. We compare this "Sample & Select" method to greedy decoding, beam search, nucleus sampling, and the recently introduced hallucination avoiding decoders of DoLA, P-CRR, and S-CRR. We show that Sample & Select improves factuality by a 30% relative margin against these decoders in NLI-based evaluation on the subsets of CNN/DM and XSum used in the FRANK benchmark, while maintaining comparable ROUGE-1 F1 scores against reference summaries. We collect human verifications of the generated summaries, confirming the factual superiority of our method.
Paper Structure (13 sections, 1 equation, 1 figure, 4 tables)

This paper contains 13 sections, 1 equation, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method: Sample and Select
  4. Experiment Setup
  5. Results and Discussion
  6. Conclusion
  7. Grammar Checking
  8. Article Cleanup
  9. Labeling Protocol
  10. Crowdworker Recruitment
  11. Metrics and Human Judgments
  12. Summary Lengths
  13. Licenses

Figures (1)

  • Figure 1: Sampling and selecting sentences iteratively.