Majority of the Bests: Improving Best-of-N via Bootstrapping

TL;DR

This paper tackles the unpredictability of Best-of-N under imperfect reward models by analyzing BoN's output distribution and identifying that the correct answer, while not highly probable, often corresponds to the mode. It introduces Majority-of-the-Bests (MoB), a bootstrapping-based method that estimates BoN's output distribution and selects the mode, with an adaptive mechanism for the subsample size $m$. The authors provide theoretical consistency results and demonstrate substantial empirical gains across five benchmarks, three base LLMs, and two reward models, improving BoN performance in 25 of 30 setups. MoB serves as a lightweight, robust alternative to BoN and Self-consistency, with potential extensions to other sampling-based strategies and practical deployment considerations.

Abstract

Sampling multiple outputs from a Large Language Model (LLM) and selecting the most frequent (Self-consistency) or highest-scoring (Best-of-N) candidate is a popular approach to achieve higher accuracy in tasks with discrete final answers. Best-of-N (BoN) selects the output with the highest reward, and with perfect rewards, it often achieves near-perfect accuracy. With imperfect rewards from reward models, however, BoN fails to reliably find the correct answer and its performance degrades drastically. We consider the distribution of BoN's outputs and highlight that, although the correct answer does not usually have a probability close to one under imperfect rewards, it is often the most likely outcome. This suggests that the mode of this distribution can be more reliably correct than a sample from it. Based on this idea, we propose Majority-of-the-Bests (MoB), a novel selection mechanism that estimates the output distribution of BoN via bootstrapping and selects its mode. Experimental results across five benchmarks, three different base LLMs, and two reward models demonstrate consistent improvements over BoN in 25 out of 30 setups. We also provide theoretical results for the consistency of the bootstrapping. MoB serves as a simple, yet strong alternative to BoN and self-consistency, and more broadly, motivates further research in more nuanced selection mechanisms.

Figures (17)

• Figure 1: Majority-of-the-Bests: first, $N$ outputs are generated for the given question. Then, we create a large number of subsets of size $m < N$ by sampling with replacement from the generated outputs. From each subset, we choose the output with the highest reward. The most frequent answer among these chosen outputs is reported as the final answer.
• Figure 2: (Left) BoN's success probability as a function of $N$ for question 647 from MMLU-Pro-Math. The success probability remains below 80%. (Middle) Distribution of the reward for correct and incorrect outputs for the same question. A separation between the two distributions is ideal. (Right) Histogram of Best-of-64 success probabilities over 500 questions.
• Figure 3: Final answer accuracy comparison of BoN, MoB, and Oracle MoB on MMLU-Pro-Math using Qwen2.5-3B (Left) and Llama3.1-8B (Right) as the base model, and ArmoRM as the reward model. Results are averaged across all problems and multiple runs. Shaded area indicates the standard error.
• Figure 4: Comparison of MoB and BoN+SC using Qwen2.5-3B as the reference model and ArmoRM as the reward model. (Left)$\pi_m$ approximation error in $\ell_1$-norm for $m=8$. (Right) Average accuracy on MMLU-Pro-Math dataset. Shaded area indicates the standard error.
• Figure 5: Comparing $m$ selection methods using ArmoRM reward model with MMLU-Pro-Math and Qwen2.5-3B (Left) and MATH500 and Llama3.1-8B (Right). Shaded area indicates the standard error.

Theorems & Definitions (5)

• Theorem 1
• Theorem 2
• proof
• Theorem 3
• proof