LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridization

TL;DR

The paper tackles the bottleneck of outcome-only reinforcement learning in retrieval-augmented reasoning by introducing LeTS, which integrates process-level rewards that supervise intermediate think-and-search steps. It defines two complementary process-level rewards—rollout-level knowledge redundancy and group-level knowledge match—and combines them with outcome-level signals via an advantage-rescaling mechanism within a GRPO-based RL framework. Empirical results across multi-hop and single-hop benchmarks show consistent performance gains and notable efficiency improvements, including reduced search steps and token generation. The work demonstrates that hybridizing process- and outcome-level rewards can substantially enhance reasoning ability and inference efficiency in LLMs, with potential applicability to broader RL settings involving structured planning.

Abstract

Large language models (LLMs) have demonstrated impressive capabilities in reasoning with the emergence of reasoning models like OpenAI-o1 and DeepSeek-R1. Recent research focuses on integrating reasoning capabilities into the realm of retrieval-augmented generation (RAG) via outcome-supervised reinforcement learning (RL) approaches, while the correctness of intermediate think-and-search steps is usually neglected. To address this issue, we design a process-level reward module to mitigate the unawareness of intermediate reasoning steps in outcome-level supervision without additional annotation. Grounded on this, we propose Learning to Think-and-Search (LeTS), a novel framework that hybridizes stepwise process reward and outcome-based reward to current RL methods for RAG. Extensive experiments demonstrate the generalization and inference efficiency of LeTS across various RAG benchmarks. In addition, these results reveal the potential of process- and outcome-level reward hybridization in boosting LLMs' reasoning ability via RL under other scenarios. The code will be released soon.

Figures (6)

• Figure 1: Illustrations of redundant searching issue and irrelevant searching issue in the current learning to think-and-search methods.
• Figure 2: Collaborative comparison of performance and knowledge redundant frequency on two representative datasets with Qwen-7B-Base and Qwen-7B-Instruct as the backbone model.
• Figure 3: The core framework of LeTS extends conventional outcome-supervised GRPO by introducing a fine-grained categorization of rollouts within each group. Specifically, generated rollouts are classified as either outperforming or underperforming, based on their relative returns. Separate reward functions are then applied to each category to compute process-level rewards. Finally, the rollout-level advantages are rescaled at the step level using an advantage normalization procedure, enabling more precise credit assignment during policy updates.
• Figure 4: Training reward during training for the four methods.
• Figure 5: Knowledge redundancy frequency of the four methods during training.