Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning

Fengran Mo, Yifan Gao, Sha Li, Hansi Zeng, Xin Liu, Zhaoxuan Tan, Xian Li, Jianshu Chen, Dakuo Wang, Meng Jiang

TL;DR

The paper addresses multi-turn conversational search where user intent evolves with history, proposing a contextualized, RL-based agentic framework that interleaves search and reasoning across turns. It introduces a decomposed reward scheme—Outcome Generation, Information Gain, and Mixed-Initiative Action—optimizing via Group Relative Policy Optimization to learn history-conditioned query generation and effective use of retrieved results. Empirical results across four benchmarks show consistent gains in both answer quality and retrieval performance, often surpassing strong baselines and even some proprietary LLMs. The work demonstrates the practical value of integrating search, reasoning, and mixed-initiative behavior in dialogue systems, while noting computation efficiency and generalization as avenues for future improvement.

Abstract

Large Language Models (LLMs) have become a popular interface for human-AI interaction, supporting information seeking and task assistance through natural, multi-turn dialogue. To respond to users within multi-turn dialogues, the context-dependent user intent evolves across interactions, requiring contextual interpretation, query reformulation, and dynamic coordination between retrieval and generation. Existing studies usually follow static rewrite, retrieve, and generate pipelines, which optimize different procedures separately and overlook the mixed-initiative action optimization simultaneously. Although the recent developments in deep search agents demonstrate the effectiveness in jointly optimizing retrieval and generation via reasoning, these approaches focus on single-turn scenarios, which might lack the ability to handle multi-turn interactions. We introduce a conversational agent that interleaves search and reasoning across turns, enabling exploratory and adaptive behaviors learned through reinforcement learning (RL) training with tailored rewards towards evolving user goals. The experimental results across four widely used conversational benchmarks demonstrate the effectiveness of our methods by surpassing several existing strong baselines.

Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning

TL;DR

The paper addresses multi-turn conversational search where user intent evolves with history, proposing a contextualized, RL-based agentic framework that interleaves search and reasoning across turns. It introduces a decomposed reward scheme—Outcome Generation, Information Gain, and Mixed-Initiative Action—optimizing via Group Relative Policy Optimization to learn history-conditioned query generation and effective use of retrieved results. Empirical results across four benchmarks show consistent gains in both answer quality and retrieval performance, often surpassing strong baselines and even some proprietary LLMs. The work demonstrates the practical value of integrating search, reasoning, and mixed-initiative behavior in dialogue systems, while noting computation efficiency and generalization as avenues for future improvement.

Abstract

Large Language Models (LLMs) have become a popular interface for human-AI interaction, supporting information seeking and task assistance through natural, multi-turn dialogue. To respond to users within multi-turn dialogues, the context-dependent user intent evolves across interactions, requiring contextual interpretation, query reformulation, and dynamic coordination between retrieval and generation. Existing studies usually follow static rewrite, retrieve, and generate pipelines, which optimize different procedures separately and overlook the mixed-initiative action optimization simultaneously. Although the recent developments in deep search agents demonstrate the effectiveness in jointly optimizing retrieval and generation via reasoning, these approaches focus on single-turn scenarios, which might lack the ability to handle multi-turn interactions. We introduce a conversational agent that interleaves search and reasoning across turns, enabling exploratory and adaptive behaviors learned through reinforcement learning (RL) training with tailored rewards towards evolving user goals. The experimental results across four widely used conversational benchmarks demonstrate the effectiveness of our methods by surpassing several existing strong baselines.
Paper Structure (23 sections, 5 equations, 3 figures, 6 tables)

This paper contains 23 sections, 5 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Task Formulation
  5. Rewards Modeling
  6. Outcome Generation Reward
  7. Information Gain Reward
  8. Mixed-Initiative Action Reward
  9. Contextualized Reasoning in Conversations with RL
  10. Usage of Clarification Results
  11. Experiments
  12. Experimental Setup
  13. Main Results
  14. Answer Generation Performance
  15. Search Results Performance
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Overview of our designed agentic conversational search framework. The GRPO Optimization is guided by full trajectories sampled from the policy LLM, which consists of reasoning, search, mix-initiative action, and answer steps, with rewards assigned to each signal. The policy model interleaves the same action set while interacting with a search environment within the Interaction Loop for both training and inference.
  • Figure 2: Correlation between search quality and answer generation performance on two main competitors and our variants of w/. or w/o information gain reward.
  • Figure 3: Impact of conversational context on the reasoning effectiveness and efficiency with different models.