Contextualizing Search Queries In-Context Learning for Conversational Rewriting with LLMs

TL;DR

The paper tackles the challenge of rewriting context-dependent conversational queries into standalone forms in low-resource settings by proposing Prompt-Guided In-Context Learning, which leverages carefully designed prompts and a small set of demonstrations to elicit context-aware rewrites from pre-trained LLMs without fine-tuning. It details a structured prompt architecture, including task definition, input/output formats, illustrative examples, and a defined test input, to guide generation. Empirical results on TREC and Taskmaster-1 show consistent improvements across BLEU-4, ROUGE-L, Success Rate@10, and MRR over strong baselines and ablations confirm the value of in-context examples; human evaluation further validates fluency, relevance, and context use. The work demonstrates a data-efficient, scalable approach to conversational search that reduces labeling and training requirements while maintaining high-quality rewrites and downstream retrieval performance.

Abstract

Conversational query rewriting is crucial for effective conversational search, yet traditional supervised methods require substantial labeled data, which is scarce in low-resource settings. This paper introduces Prompt-Guided In-Context Learning, a novel approach that leverages the in-context learning capabilities of Large Language Models (LLMs) for few-shot conversational query rewriting. Our method employs carefully designed prompts, incorporating task descriptions, input/output format specifications, and a small set of illustrative examples, to guide pre-trained LLMs to generate context-independent queries without explicit fine-tuning. Extensive experiments on benchmark datasets, TREC and Taskmaster-1, demonstrate that our approach significantly outperforms strong baselines, including supervised models and contrastive co-training methods, across various evaluation metrics such as BLEU, ROUGE-L, Success Rate, and MRR. Ablation studies confirm the importance of in-context examples, and human evaluations further validate the superior fluency, relevance, and context utilization of our generated rewrites. The results highlight the potential of prompt-guided in-context learning as an efficient and effective paradigm for low-resource conversational query rewriting, reducing the reliance on extensive labeled data and complex training procedures.