Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DialogueReason: Rule-Based RL Sparks Dialogue Reasoning in LLMs

Yubo Shu, Zhewei Huang, Xin Wu, Chen Hu, Shuchang Zhou, Daxin Jiang

TL;DR

The paper addresses the limitations of monologue-style reasoning in RL-powered large language models, notably reduced diversity and coherency when solving compound problems. It introduces DialogueReason, a dialogue-based reasoning pattern trained via PPO with rule-based rewards, enabling multiple agents and environments to orchestrate diverse yet coherent reasoning. To stress-test reasoning, it defines Compound-QA, concatenating multiple solvable subproblems and evaluating performance on MATH, AIME, and GPQA, where DialogueReason consistently outperforms monologue baselines as complexity rises. The work highlights improved interpretability, controllability, and potential for scalable multi-agent training, while outlining future directions to broaden domains and disentangle evaluation metrics for reasoning diversity and coherency.

Abstract

We propose DialogueReason, a reasoning paradigm that uncovers the lost roles in monologue-style reasoning models, aiming to boost diversity and coherency of the reasoning process. Recent advances in RL-based large reasoning models have led to impressive long CoT capabilities and high performance on math and science benchmarks. However, these reasoning models rely mainly on monologue-style reasoning, which often limits reasoning diversity and coherency, frequently recycling fixed strategies or exhibiting unnecessary shifts in attention. Our work consists of an analysis of monologue reasoning patterns and the development of a dialogue-based reasoning approach. We first introduce the Compound-QA task, which concatenates multiple problems into a single prompt to assess both diversity and coherency of reasoning. Our analysis shows that Compound-QA exposes weaknesses in monologue reasoning, evidenced by both quantitative metrics and qualitative reasoning traces. Building on the analysis, we propose a dialogue-based reasoning, named DialogueReason, structured around agents, environment, and interactions. Using PPO with rule-based rewards, we train open-source LLMs (Qwen-QWQ and Qwen-Base) to adopt dialogue reasoning. We evaluate trained models on MATH, AIME, and GPQA datasets, showing that the dialogue reasoning model outperforms monologue models under more complex compound questions. Additionally, we discuss how dialogue-based reasoning helps enhance interpretability, facilitate more intuitive human interaction, and inspire advances in multi-agent system design.

DialogueReason: Rule-Based RL Sparks Dialogue Reasoning in LLMs

TL;DR

The paper addresses the limitations of monologue-style reasoning in RL-powered large language models, notably reduced diversity and coherency when solving compound problems. It introduces DialogueReason, a dialogue-based reasoning pattern trained via PPO with rule-based rewards, enabling multiple agents and environments to orchestrate diverse yet coherent reasoning. To stress-test reasoning, it defines Compound-QA, concatenating multiple solvable subproblems and evaluating performance on MATH, AIME, and GPQA, where DialogueReason consistently outperforms monologue baselines as complexity rises. The work highlights improved interpretability, controllability, and potential for scalable multi-agent training, while outlining future directions to broaden domains and disentangle evaluation metrics for reasoning diversity and coherency.

Abstract

We propose DialogueReason, a reasoning paradigm that uncovers the lost roles in monologue-style reasoning models, aiming to boost diversity and coherency of the reasoning process. Recent advances in RL-based large reasoning models have led to impressive long CoT capabilities and high performance on math and science benchmarks. However, these reasoning models rely mainly on monologue-style reasoning, which often limits reasoning diversity and coherency, frequently recycling fixed strategies or exhibiting unnecessary shifts in attention. Our work consists of an analysis of monologue reasoning patterns and the development of a dialogue-based reasoning approach. We first introduce the Compound-QA task, which concatenates multiple problems into a single prompt to assess both diversity and coherency of reasoning. Our analysis shows that Compound-QA exposes weaknesses in monologue reasoning, evidenced by both quantitative metrics and qualitative reasoning traces. Building on the analysis, we propose a dialogue-based reasoning, named DialogueReason, structured around agents, environment, and interactions. Using PPO with rule-based rewards, we train open-source LLMs (Qwen-QWQ and Qwen-Base) to adopt dialogue reasoning. We evaluate trained models on MATH, AIME, and GPQA datasets, showing that the dialogue reasoning model outperforms monologue models under more complex compound questions. Additionally, we discuss how dialogue-based reasoning helps enhance interpretability, facilitate more intuitive human interaction, and inspire advances in multi-agent system design.
Paper Structure (29 sections, 7 equations, 7 figures, 2 tables)

This paper contains 29 sections, 7 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Monologue Reasoning Inspection
  3. Diversity and Coherency in Thinking
  4. Compound-QA Task
  5. Compound-QA Task Definition.
  6. Results
  7. Analysis
  8. Inability to deepen reasoning: attention deficit.
  9. Only answering the first question.
  10. Hard to balance coherency and diversity.
  11. DialogueReason
  12. Reasoning Pattern Perspective: Dialogue-Based Reasoning
  13. Rule-based RL training to adopt the dialogue pattern.
  14. Results
  15. Analysis
  16. ...and 14 more sections

Figures (7)

  • Figure 1: Illustration of the reasoning process as an interplay between divergent and convergent thinking.
  • Figure 2: Illustration of the Compound-QA task, which involves reasoning over multiple sub-questions concatenated into a single prompt. The figure also highlights the relationship between Compound-QA and Single-QA.
  • Figure 3: Three representative failure made by monologue reasoning models when tackling compound questions.
  • Figure 4: The illustration of the dialogue-based reasoning pattern.
  • Figure 5: The four subplots correspond to Overall (a), MATH‑500 (b), AIME24 (c), and GPQA-Diamond (d), respectively. The x-axis represents the compound question size from 1 to 10, while the y-axis shows the Remain Rate — the accuracy at each cbK divided by the accuracy at cbK1. A value closer to 1 indicates more stable performance.
  • ...and 2 more figures