Ranked Voting based Self-Consistency of Large Language Models

TL;DR

The paper tackles the limitation of single-answer self-consistency in large language models by proposing ranked answers and applying ranked voting (IRV, BCV, MRRV) to aggregate multiple reasoning outputs. It formalizes the ranked-output framework, constructs ranked demonstrations, and evaluates on six diverse QA datasets with both open-source and closed-source LLMs, showing consistent performance gains over traditional majority-vote baselines. Key contributions include three voting schemes, a robust few-shot example construction strategy, and empirical evidence that ranking information improves reliability and reduces ties, with practical implications for improving LLM reasoning. The approach offers a scalable enhancement to self-consistency that can be integrated with existing CoT prompting techniques, potentially broadening the reliability of LLM-assisted reasoning in real-world tasks.

Abstract

Majority voting is considered an effective method to enhance chain-of-thought reasoning, as it selects the answer with the highest "self-consistency" among different reasoning paths (Wang et al., 2023). However, previous chain-of-thought reasoning methods typically generate only a single answer in each trial, thereby ignoring the possibility of other potential answers. As a result, these alternative answers are often overlooked in subsequent voting processes. In this work, we propose to generate ranked answers in each reasoning process and conduct ranked voting among multiple ranked answers from different responses, thereby making the overall self-consistency more reliable. Specifically, we use three ranked voting methods: Instant-runoff voting, Borda count voting, and mean reciprocal rank voting. We validate our methods on six datasets, including three multiple-choice and three open-ended question-answering tasks, using both advanced open-source and closed-source large language models. Extensive experimental results indicate that our proposed method outperforms the baselines, showcasing the potential of leveraging the information of ranked answers and using ranked voting to improve reasoning performance. The code is available at https://github.com/szu-tera/RankedVotingSC.

Figures (6)

• Figure 1: Example of (a) majority voting based self-consistency among single answers wang2023selfconsistency and (b) ranked answers in four responses by models.
• Figure 2: Examples of the procedures for three ranked voting methods. (a) The obtained ranked answers ($k=8$). (b) Instant-runoff voting. (c) Borda count voting. (d) Mean reciprocal rank voting.
• Figure 3: Comparison of methods with different number of responses $k$.
• Figure 4: Comparison of methods with different shuffled examples for few-shot learning.
• Figure 5: The impact of ranked answers (in blue dashed box) and ranked voting (in red dashed box). Ranked voting is more effective and capable of delivering substantial and reliable improvements.