Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design

Xu Guo, Qiming Ge, Jian Tong, Kedi Chen, Jin Zhang, Xiaogui Yang, Xuan Gao, Haijun Lv, Zhihui Lu, Yicheng Zou, Qipeng Guo

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the reasoning capabilities of Large Language Models. When applied to RLVR, Multiple-Choice Questions (MCQs) offer a scalable source of verifiable data but risk inducing reward hacking, where models shortcut reasoning via random guessing or simple elimination. Current approaches often mitigate this by converting MCQs to open-ended formats, thereby discarding the contrastive signal provided by expert-designed distractors. In this work, we systematically investigate the impact of option design on RLVR. Our analysis highlights two primary insights: (1) Mismatches in option counts between training and testing degrade performance. (2) Strong distractors effectively mitigate random guessing, enabling effective RLVR training even with 2-way questions. Motivated by these findings, we propose Iterative Distractor Curation (IDC), a framework that actively constructs high-quality distractors to block elimination shortcuts and promote deep reasoning. Experiments on various benchmarks demonstrate that our method effectively enhances distractor quality and yields significant gains in RLVR training compared to the original data.

Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the reasoning capabilities of Large Language Models. When applied to RLVR, Multiple-Choice Questions (MCQs) offer a scalable source of verifiable data but risk inducing reward hacking, where models shortcut reasoning via random guessing or simple elimination. Current approaches often mitigate this by converting MCQs to open-ended formats, thereby discarding the contrastive signal provided by expert-designed distractors. In this work, we systematically investigate the impact of option design on RLVR. Our analysis highlights two primary insights: (1) Mismatches in option counts between training and testing degrade performance. (2) Strong distractors effectively mitigate random guessing, enabling effective RLVR training even with 2-way questions. Motivated by these findings, we propose Iterative Distractor Curation (IDC), a framework that actively constructs high-quality distractors to block elimination shortcuts and promote deep reasoning. Experiments on various benchmarks demonstrate that our method effectively enhances distractor quality and yields significant gains in RLVR training compared to the original data.
Paper Structure (44 sections, 7 equations, 6 figures, 14 tables)

This paper contains 44 sections, 7 equations, 6 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning with Verifiable Rewards (RLVR).
  4. Hard Negative Samples and Preference Learning.
  5. LLM-based Data Generation.
  6. Preliminaries
  7. Impact of the Number of Distractors
  8. Analysis Study
  9. Aligning Train-Test Formats Benefits RLVR Training
  10. The Effect of Distractor Strength
  11. Analysis Study
  12. Iterative Distractor Curation
  13. Experimental Setup
  14. Results
  15. Ablation Studies
  16. ...and 29 more sections

Figures (6)

  • Figure 1: The impact of distractor design on RLVR.
  • Figure 2: Relationship between average normalized $z$ score and option-count gap $\Delta$. Note that data points are slightly offset horizontally for clarity.
  • Figure 3: Training dynamics analysis: response length (top row) and solve-all ratio (bottom row) for Llama (left column) and Qwen (right column) across distractor settings. Removing strong distractors (w/o distractor) leads to rapid saturation and shorter responses.
  • Figure 4: The framework of Iterative Distractor Curation (IDC).
  • Figure 5: Performance gains and distractor dynamics across rejection sampling rounds.
  • ...and 1 more figures