Diversity-Enhanced Reasoning for Subjective Questions

TL;DR

This work tackles the challenge that reinforcement-learning with verifiable rewards (RLVR) degrades generation diversity, hindering subjective reasoning tasks with multiple valid outcomes. It introduces MultiRole-R1, a two-stage framework that (i) synthesizes multi-role reasoning paths to inject perspective diversity and (ii) applies diversity-enhanced reinforcement learning via Generalized Reward Policy Optimization with a diversity shaping reward, promoting broader CoT exploration. Empirically, MultiRole-R1 achieves substantial gains on in-domain subjective tasks (≈14.1% accuracy) and out-of-domain tasks (≈7.64%), with transfer to advanced math reasoning (AIME 2024) and a strong per-task diversity-accuracy correlation (r ≈ 0.74). The results demonstrate that perspective diversity is a primary performance driver and that token-level diversity aids test-time scaling, suggesting that diversity-focused CoT can outperform longer reasoning chains. Overall, the framework enables robust, generalizable subjective reasoning and highlights diversity as a more reliable accuracy proxy than reasoning length.

Abstract

Large Reasoning Models (LRMs) with long chain-of-thought capabilities, optimized via reinforcement learning with verifiable rewards (RLVR), excel at objective reasoning tasks like mathematical problem solving and code generation. However, RLVR is known for degrading generation diversity, which causes LRMs to fall short on subjective reasoning that has multiple answers depending on different role perspectives. While recent studies recognize the importance of diversity-enhanced training in objective reasoning, limited attention has been given to subjective tasks. In this paper, we find that subjective reasoning can be improved by introducing perspective diversity and token-level diversity, with the former one providing a coherent scaffolding anchored to a real-world stakeholder group and the latter one broadening the answer search space. We propose MultiRole-R1, a diversity-enhanced training framework featuring an unsupervised data construction pipeline that synthesizes reasoning chains incorporating various role perspectives. It also employs reinforcement learning via Group Relative Policy Optimization with reward shaping, taking diversity as a reward signal in addition to verifiable reward. Training on subjective tasks solely, MultiRole-R1 increases the in-domain and out-of-domain accuracy by 14.1% and 7.64%, and even enhances the performance on advanced math reasoning such as AIME 2024. We further show that diversity is a more consistent indicator of accuracy than reasoning length.

Figures (7)

• Figure 1: Illustration of MultiRole-R1 framework. Stage 1 (enhance perspective diversity): LRMs generate seed roles with contrastive opinions, and sample diverse reasoning paths from different roles. We concatenate paths from different perspectives into a CoT, and then finetune the model to follow the multi-role reasoning format. Stage 2 (enhance token-level diversity): we utilize GRPO with diversity reward shaping. Verifiable rewards are applied depending on whether the ground-truth varies by roles. We take diversity as an additional reward to promote exploration efficiency.
• Figure 2: (a) The performance of Deepseek-R1-Distill-Qwen-7B deepseekr1 under different reasoning length settings across different datasets. The bar chart shows that longer reasoning chains result in higher accuracy on subjective tasks. (b) Accuracy gain of Deepseek-R1-Distill-Qwen-7B deepseekr1 when trailing with more wait tokens. (c) Demonstration of the number of distinct opinions increases as more roles are involved in a single reasoning chain.
• Figure 3: Qualitative example of the 32 most frequent roles in the training data generated by LRMs.
• Figure 4: Comparison of Pass@k accuracy of R1-Distill-Qwen-7B on GLOQA dataset, w/ and w/o diversity reward shaping.
• Figure 5: We present the top 100 most frequent roles from the model output during test-time. The diameters of the circles are proportional to the frequency.