ChatR1: Reinforcement Learning for Conversational Reasoning and Retrieval Augmented Question Answering

TL;DR

ChatR1 introduces a reinforcement-learning framework for conversational question answering that interleaves reasoning and retrieval across dialogue turns to handle evolving user intents. It defines a composite reward $R(\\tau) = R_{answer}(y) + \\alpha R_{intent}(Q)$ and optimizes with Proximal Policy Optimization to align intermediate decisions with user goals. Empirically, ChatR1 with 3B and 7B backbones outperforms SFT and RL baselines across five diverse CQA datasets, with ablations confirming the value of the intent reward and analyses showing diverse reasoning trajectories and effective use of the search tool. The results demonstrate robust cross-domain generalization and underscore the potential of RL-based conversational reasoning to improve grounding and adaptivity in real-world multi-turn search tasks.

Abstract

We present ChatR1, a reasoning framework based on reinforcement learning (RL) for conversational question answering (CQA). Reasoning plays an important role in CQA, where user intent evolves across dialogue turns, and utterances are often underspecified, requiring contextual interpretation, query reformulation, and dynamic coordination between retrieval and generation. Unlike static `rewrite, retrieve, and generate' pipelines, ChatR1 interleaves search and reasoning across turns, enabling exploratory and adaptive behaviors learned through RL. To address the challenge of sparse and delayed rewards in RL, we propose an intent-aware reward that provides turn-level feedback by aligning retrieval and reasoning with evolving user goals. Our proposed ChatR1 demonstrates strong performance on both 3B and 7B model backbones, outperforming competitive models on five CQA datasets, measured by different metrics (F1, BERTScore, and LLM-as-judge). We include a diverse set of CQA datasets to cover topic shifts, evolving intents, mixed-initiative dialogues, and multi-document grounding, testing ChatR1's performance from various aspects. Ablation studies confirm the effectiveness of the intent-aware reward. Our analyses further reveal diverse reasoning trajectories and effective use of the search tool. ChatR1 also generalizes robustly across domains, demonstrating that RL-based reasoning enables more flexible and context-sensitive behavior than static CQA pipelines.

Figures (9)

• Figure 1: Reasoning in CQA requires the system to understand the user intent. The system resolves anaphoras and ellipses in the last user utterance ($\textbf{U2}$) during its internal reasoning, and formulates a search query, before producing the final answer.
• Figure 2: Overview of ChatR1's interaction loop: the policy model interleaves reasoning, search, and answering while interacting with a search environment. (Right) Full trajectories are sampled from the policy LLM and consist of all reasoning, search, and answer steps, with rewards assigned to search queries (intent-aware reward) and the final answer. These signals jointly train the policy under PPO optimization.
• Figure 3: Out-of-domain transfer performances for models trained on QReCC, with in-domain reference.
• Figure 4:
• Figure 5: (Left) Analysis of intent rewards, contrasting passage-level rewards (hit@k) with query-level rewards (F1), and their impact on conversational QA response quality. (Right) Relationship between hit@k and query-level F1. Both on TopiOCQA with ChatR1-3B.