Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When should I search more: Adaptive Complex Query Optimization with Reinforcement Learning

Wei Wen, Sihang Deng, Tianjun Wei, Keyu Chen, Ruizhi Qiao, Xing Sun

TL;DR

This work tackles the challenge of complex query optimization in Retrieval-Augmented Generation by introducing Adaptive Complex Query Optimization (ACQO), an RL framework that adaptively decides when to decompose or disambiguate queries (via Adaptive Query Reformulation, AQR) and robustly combines multi-retriever results (via Rank-Score Fusion, RSF). It employs a two-stage Curriculum Reinforcement Learning (CRL) to stabilize training, first exploring broad decomposition strategies and then converging on high-quality, retriever-specific query schemes. Empirical results show state-of-the-art performance on TopiOCQA and HotpotQA, with strong generalization to unseen domains (MultiHop-RAG) and significantly lower latency compared to competitive baselines. The approach demonstrates that retriever-aware, adaptive query strategies can substantially improve both retrieval metrics and end-to-end QA quality in production-ready RAG systems.

Abstract

Query optimization is a crucial component for the efficacy of Retrieval-Augmented Generation (RAG) systems. While reinforcement learning (RL)-based agentic and reasoning methods have recently emerged as a promising direction on query optimization, most existing approaches focus on the expansion and abstraction of a single query. However, complex user queries are prevalent in real-world scenarios, often requiring multiple parallel and sequential search strategies to handle disambiguation and decomposition. Directly applying RL to these complex cases introduces significant hurdles. Determining the optimal number of sub-queries and effectively re-ranking and merging retrieved documents vastly expands the search space and complicates reward design, frequently leading to training instability. To address these challenges, we propose a novel RL framework called Adaptive Complex Query Optimization (ACQO). Our framework is designed to adaptively determine when and how to expand the search process. It features two core components: an Adaptive Query Reformulation (AQR) module that dynamically decides when to decompose a query into multiple sub-queries, and a Rank-Score Fusion (RSF) module that ensures robust result aggregation and provides stable reward signals for the learning agent. To mitigate training instabilities, we adopt a Curriculum Reinforcement Learning (CRL) approach, which stabilizes the training process by progressively introducing more challenging queries through a two-stage strategy. Our comprehensive experiments demonstrate that ACQO achieves state-of-the-art performance on three complex query benchmarks, significantly outperforming established baselines. The framework also showcases improved computational efficiency and broad compatibility with different retrieval architectures, establishing it as a powerful and generalizable solution for next-generation RAG systems.

When should I search more: Adaptive Complex Query Optimization with Reinforcement Learning

TL;DR

This work tackles the challenge of complex query optimization in Retrieval-Augmented Generation by introducing Adaptive Complex Query Optimization (ACQO), an RL framework that adaptively decides when to decompose or disambiguate queries (via Adaptive Query Reformulation, AQR) and robustly combines multi-retriever results (via Rank-Score Fusion, RSF). It employs a two-stage Curriculum Reinforcement Learning (CRL) to stabilize training, first exploring broad decomposition strategies and then converging on high-quality, retriever-specific query schemes. Empirical results show state-of-the-art performance on TopiOCQA and HotpotQA, with strong generalization to unseen domains (MultiHop-RAG) and significantly lower latency compared to competitive baselines. The approach demonstrates that retriever-aware, adaptive query strategies can substantially improve both retrieval metrics and end-to-end QA quality in production-ready RAG systems.

Abstract

Query optimization is a crucial component for the efficacy of Retrieval-Augmented Generation (RAG) systems. While reinforcement learning (RL)-based agentic and reasoning methods have recently emerged as a promising direction on query optimization, most existing approaches focus on the expansion and abstraction of a single query. However, complex user queries are prevalent in real-world scenarios, often requiring multiple parallel and sequential search strategies to handle disambiguation and decomposition. Directly applying RL to these complex cases introduces significant hurdles. Determining the optimal number of sub-queries and effectively re-ranking and merging retrieved documents vastly expands the search space and complicates reward design, frequently leading to training instability. To address these challenges, we propose a novel RL framework called Adaptive Complex Query Optimization (ACQO). Our framework is designed to adaptively determine when and how to expand the search process. It features two core components: an Adaptive Query Reformulation (AQR) module that dynamically decides when to decompose a query into multiple sub-queries, and a Rank-Score Fusion (RSF) module that ensures robust result aggregation and provides stable reward signals for the learning agent. To mitigate training instabilities, we adopt a Curriculum Reinforcement Learning (CRL) approach, which stabilizes the training process by progressively introducing more challenging queries through a two-stage strategy. Our comprehensive experiments demonstrate that ACQO achieves state-of-the-art performance on three complex query benchmarks, significantly outperforming established baselines. The framework also showcases improved computational efficiency and broad compatibility with different retrieval architectures, establishing it as a powerful and generalizable solution for next-generation RAG systems.
Paper Structure (52 sections, 10 equations, 7 figures, 11 tables)

This paper contains 52 sections, 10 equations, 7 figures, 11 tables.

Table of Contents

  1. Introduction
  2. What makes queries complex in real-world RAG scenarios?
  3. Query Complexity Analysis Framework
  4. Dataset Analysis: Prevalence of Complex Queries
  5. Why Current Methods Struggle with Complex Queries
  6. Adaptive Complex Query Optimization
  7. Task Formulation
  8. Overall Framework
  9. Re-ranker Design
  10. Method.
  11. Advantages.
  12. Curriculum Reinforcement Learning
  13. Base Reward Function
  14. Stage I: Explore-Oriented CRL
  15. Data Curriculum.
  16. ...and 37 more sections

Figures (7)

  • Figure 1: Distribution of query complexity.
  • Figure 2: Performance with different query counts.
  • Figure 3: Overview of ACQO. ACQO employs two-stage curriculum reinforcement learning to adaptively optimize complex queries and integrate multi-retrieval results via Rank-Score Fusion.
  • Figure 4: Query adaptation and performance improvement on TopiOCQA(L) and HotpotQA(R).
  • Figure 5: Comparison of queries generated by models trained with different retrievers and their retrieval performance across different retrievers.
  • ...and 2 more figures