Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DARL: Encouraging Diverse Answers for General Reasoning without Verifiers

Chongxuan Huang, Lei Lin, Xiaodong Shi, Wenping Hu, Ruiming Tang

TL;DR

DARL addresses the overfitting to reference answers in verifier-based and verifier-free RL for general reasoning. It introduces a diversity-aware reward that allows outputs to deviate within a bounded range from the reference while maintaining alignment, implemented via a dynamic thresholding scheme. The approach is verifier-free, compatible with existing RL methods, and demonstrates consistent gains across diverse reasoning and general-domain benchmarks, especially for tasks with multiple valid reasoning paths. The results highlight improved output diversity, higher policy entropy, and robust generalization without domain-specific verifiers, enabling more flexible, open-ended reasoning capabilities.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated promising gains in enhancing the reasoning capabilities of large language models. However, its dependence on domain-specific verifiers significantly restricts its applicability to open and general domains. Recent efforts such as RLPR have extended RLVR to general domains, enabling training on broader datasets and achieving improvements over RLVR. However, a notable limitation of these methods is their tendency to overfit to reference answers, which constrains the model's ability to generate diverse outputs. This limitation is particularly pronounced in open-ended tasks such as writing, where multiple plausible answers exist. To address this, we propose DARL, a simple yet effective reinforcement learning framework that encourages the generation of diverse answers within a controlled deviation range from the reference while preserving alignment with it. Our framework is fully compatible with existing general reinforcement learning methods and can be seamlessly integrated without additional verifiers. Extensive experiments on thirteen benchmarks demonstrate consistent improvements in reasoning performance. Notably, DARL surpasses RLPR, achieving average gains of 1.3 points on six reasoning benchmarks and 9.5 points on seven general benchmarks, highlighting its effectiveness in improving both reasoning accuracy and output diversity.

DARL: Encouraging Diverse Answers for General Reasoning without Verifiers

TL;DR

DARL addresses the overfitting to reference answers in verifier-based and verifier-free RL for general reasoning. It introduces a diversity-aware reward that allows outputs to deviate within a bounded range from the reference while maintaining alignment, implemented via a dynamic thresholding scheme. The approach is verifier-free, compatible with existing RL methods, and demonstrates consistent gains across diverse reasoning and general-domain benchmarks, especially for tasks with multiple valid reasoning paths. The results highlight improved output diversity, higher policy entropy, and robust generalization without domain-specific verifiers, enabling more flexible, open-ended reasoning capabilities.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated promising gains in enhancing the reasoning capabilities of large language models. However, its dependence on domain-specific verifiers significantly restricts its applicability to open and general domains. Recent efforts such as RLPR have extended RLVR to general domains, enabling training on broader datasets and achieving improvements over RLVR. However, a notable limitation of these methods is their tendency to overfit to reference answers, which constrains the model's ability to generate diverse outputs. This limitation is particularly pronounced in open-ended tasks such as writing, where multiple plausible answers exist. To address this, we propose DARL, a simple yet effective reinforcement learning framework that encourages the generation of diverse answers within a controlled deviation range from the reference while preserving alignment with it. Our framework is fully compatible with existing general reinforcement learning methods and can be seamlessly integrated without additional verifiers. Extensive experiments on thirteen benchmarks demonstrate consistent improvements in reasoning performance. Notably, DARL surpasses RLPR, achieving average gains of 1.3 points on six reasoning benchmarks and 9.5 points on seven general benchmarks, highlighting its effectiveness in improving both reasoning accuracy and output diversity.
Paper Structure (26 sections, 36 equations, 8 figures, 6 tables)

This paper contains 26 sections, 36 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language Model Reasoning.
  4. Verifier-Free Reasoning.
  5. Background
  6. Rule-based Reward.
  7. Verifier-Free Reward.
  8. Method
  9. Experimental Setup
  10. Dataset.
  11. Models.
  12. Baselines.
  13. Evaluation.
  14. Answer Diversity Evaluation.
  15. Results and Analysis
  16. ...and 11 more sections

Figures (8)

  • Figure 1: Overview of DARL. $Q$: input question, $z$: generated reasoning content before final answer, $y$: generated final answer, $y^*$: reference answer.
  • Figure 2: Policy entropy over training steps for DARL and RLPR. Our method consistently maintains higher entropy, indicating increased diversity in generated responses.
  • Figure 3: Average log-probability of ground-truth answers and alternative semantically equivalent answers during training. Compared to RLPR, DARL assigns higher likelihood to diverse yet semantically faithful answers.
  • Figure 4: Prompt used to rewrite ground-truth answers into semantically equivalent but lexically distinct variants with DeepSeek-V3.
  • Figure 5: Case Study of DARL on the WebInstruct Dataset.
  • ...and 3 more figures