Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Generative Selection for Best-of-N

Shubham Toshniwal, Aleksander Ficek, Siddhartha Jain, Wei Du, Vahid Noroozi, Sadegh Mahdavi, Somshubra Majumdar, Igor Gitman

TL;DR

This work demonstrates that small reasoning models can learn effective generative Best-of-$N$ selection through on-policy reinforcement learning, substantially improving GenSelect performance over prompting and majority voting. By constructing math and code selection tasks with automatic verifiers and nontrivial difficulty, the authors train 1.7B models with the DAPO algorithm that rival much larger models and generalize to higher-quality candidate outputs. The results show larger gains in math than in code and reveal cross-domain transfer when training on math data. Overall, reinforcement learning provides a scalable pathway for strong generative selection, enabling more efficient test-time scaling for reasoning tasks.

Abstract

Scaling test-time compute via parallel sampling can substantially improve LLM reasoning, but is often limited by Best-of-N selection quality. Generative selection methods, such as GenSelect, address this bottleneck, yet strong selection performance remains largely limited to large models. We show that small reasoning models can acquire strong GenSelect capabilities through targeted reinforcement learning. To this end, we synthesize selection tasks from large-scale math and code instruction datasets by filtering to instances with both correct and incorrect candidate solutions, and train 1.7B-parameter models with DAPO to reward correct selections. Across math (AIME24, AIME25, HMMT25) and code (LiveCodeBench) reasoning benchmarks, our models consistently outperform prompting and majority-voting baselines, often approaching or exceeding much larger models. Moreover, these gains generalize to selecting outputs from stronger models despite training only on outputs from weaker models. Overall, our results establish reinforcement learning as a scalable way to unlock strong generative selection in small models, enabling efficient test-time scaling.

Learning Generative Selection for Best-of-N

TL;DR

This work demonstrates that small reasoning models can learn effective generative Best-of- selection through on-policy reinforcement learning, substantially improving GenSelect performance over prompting and majority voting. By constructing math and code selection tasks with automatic verifiers and nontrivial difficulty, the authors train 1.7B models with the DAPO algorithm that rival much larger models and generalize to higher-quality candidate outputs. The results show larger gains in math than in code and reveal cross-domain transfer when training on math data. Overall, reinforcement learning provides a scalable pathway for strong generative selection, enabling more efficient test-time scaling for reasoning tasks.

Abstract

Scaling test-time compute via parallel sampling can substantially improve LLM reasoning, but is often limited by Best-of-N selection quality. Generative selection methods, such as GenSelect, address this bottleneck, yet strong selection performance remains largely limited to large models. We show that small reasoning models can acquire strong GenSelect capabilities through targeted reinforcement learning. To this end, we synthesize selection tasks from large-scale math and code instruction datasets by filtering to instances with both correct and incorrect candidate solutions, and train 1.7B-parameter models with DAPO to reward correct selections. Across math (AIME24, AIME25, HMMT25) and code (LiveCodeBench) reasoning benchmarks, our models consistently outperform prompting and majority-voting baselines, often approaching or exceeding much larger models. Moreover, these gains generalize to selecting outputs from stronger models despite training only on outputs from weaker models. Overall, our results establish reinforcement learning as a scalable way to unlock strong generative selection in small models, enabling efficient test-time scaling.
Paper Structure (15 sections, 3 figures, 1 table)

This paper contains 15 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Data Construction
  3. Math
  4. Code
  5. Experimental Setup
  6. RL Training
  7. Evaluation
  8. Results
  9. Results on Qwen3-1.7B Generations
  10. Results on Higher Quality Generations
  11. Math and Coding Model Differences
  12. Related Work
  13. Reward Models
  14. Parallel Reasoning Selection
  15. Conclusion

Figures (3)

  • Figure 1: Accuracy of Qwen3-1.7B generations when selected using different strategies on math and coding benchmarks; RL-trained GenSelect consistently outperforms prompting and majority voting.
  • Figure 2: The GenSelect prompt used for selection in prompt baselines and RL training.
  • Figure 3: RL training dynamics of Qwen3-1.7B for GenSelect on math (left) and code (right) selection tasks. Selection accuracy improves rapidly and stabilizes for math, while training on code shows noisier dynamics and earlier saturation, reflecting domain-dependent differences in selection difficulty.