Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking On-policy Optimization for Query Augmentation

Zhichao Xu, Shengyao Zhuang, Xueguang Ma, Bingsen Chen, Yijun Tian, Fengran Mo, Jie Cao, Vivek Srikumar

TL;DR

Rethinking On-policy Optimization for Query Augmentation compares prompting-based and RL-based query augmentation for IR across diverse benchmarks. The study finds that simple zero-shot prompting with pseudo-documents often matches or surpasses RL-based rewriting, motivating a hybrid approach OPQE that optimizes pseudo-documents with retrieval rewards. OPQE delivers the best overall performance, especially for dense retrievers, while SPQE remains a robust and efficient baseline; the work includes tool retrieval evaluation and highlights practical implications for API-based or black-box retrievers. The authors provide reproducible code and discuss future directions for prompting strategies and reward design.

Abstract

Recent advances in large language models (LLMs) have led to a surge of interest in query augmentation for information retrieval (IR). Two main approaches have emerged. The first prompts LLMs to generate answers or pseudo-documents that serve as new queries, relying purely on the model's parametric knowledge or contextual information. The second applies reinforcement learning (RL) to fine-tune LLMs for query rewriting, directly optimizing retrieval metrics. While having respective advantages and limitations, the two approaches have not been compared under consistent experimental conditions. In this work, we present the first systematic comparison of prompting-based and RL-based query augmentation across diverse benchmarks, including evidence-seeking, ad hoc, and tool retrieval. Our key finding is that simple, training-free query augmentation often performs on par with, or even surpasses, more expensive RL-based counterparts, especially when using powerful LLMs. Motivated by this discovery, we introduce a novel hybrid method, On-policy Pseudo-document Query Expansion (OPQE), which, instead of rewriting a query, the LLM policy learns to generate a pseudo-document that maximizes retrieval performance, thus merging the flexibility and generative structure of prompting with the targeted optimization of RL. We show OPQE outperforms both standalone prompting and RL-based rewriting, demonstrating that a synergistic approach yields the best results. Our implementation is made available to facilitate reproducibility.

Rethinking On-policy Optimization for Query Augmentation

TL;DR

Rethinking On-policy Optimization for Query Augmentation compares prompting-based and RL-based query augmentation for IR across diverse benchmarks. The study finds that simple zero-shot prompting with pseudo-documents often matches or surpasses RL-based rewriting, motivating a hybrid approach OPQE that optimizes pseudo-documents with retrieval rewards. OPQE delivers the best overall performance, especially for dense retrievers, while SPQE remains a robust and efficient baseline; the work includes tool retrieval evaluation and highlights practical implications for API-based or black-box retrievers. The authors provide reproducible code and discuss future directions for prompting strategies and reward design.

Abstract

Recent advances in large language models (LLMs) have led to a surge of interest in query augmentation for information retrieval (IR). Two main approaches have emerged. The first prompts LLMs to generate answers or pseudo-documents that serve as new queries, relying purely on the model's parametric knowledge or contextual information. The second applies reinforcement learning (RL) to fine-tune LLMs for query rewriting, directly optimizing retrieval metrics. While having respective advantages and limitations, the two approaches have not been compared under consistent experimental conditions. In this work, we present the first systematic comparison of prompting-based and RL-based query augmentation across diverse benchmarks, including evidence-seeking, ad hoc, and tool retrieval. Our key finding is that simple, training-free query augmentation often performs on par with, or even surpasses, more expensive RL-based counterparts, especially when using powerful LLMs. Motivated by this discovery, we introduce a novel hybrid method, On-policy Pseudo-document Query Expansion (OPQE), which, instead of rewriting a query, the LLM policy learns to generate a pseudo-document that maximizes retrieval performance, thus merging the flexibility and generative structure of prompting with the targeted optimization of RL. We show OPQE outperforms both standalone prompting and RL-based rewriting, demonstrating that a synergistic approach yields the best results. Our implementation is made available to facilitate reproducibility.
Paper Structure (18 sections, 4 equations, 5 figures, 8 tables, 1 algorithm)

This paper contains 18 sections, 4 equations, 5 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Background
  4. Comparison
  5. Methodology
  6. Experimental Setup
  7. Result and Analysis
  8. Extending to Tool Retrieval
  9. Discussions
  10. On-policy Pseudo-document Query Expansion
  11. Method
  12. Experiment, Result and Analysis
  13. Conclusion and Future Works
  14. Dataset Statistics and Licenses
  15. Results of SPQE with Different LLMs
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Reward curves of OPQE vs. RL on FEVER and TriviaQA datasets across different retrievers.
  • Figure 2: The RL (DeepRetrieval) prompt template for NQ dataset --- sparse retrieval. Notice the boolean operators.
  • Figure 3: The RL (DeepRetrieval) prompt template for NQ dataset --- dense retrieval.
  • Figure 4: SPQE prompt template for NQ dataset. We use same prompt template for sparse and dense retrieval.
  • Figure 5: The OPQE prompt template for NQ dataset. We use same prompt template for sparse and dense retrieval.