Passage Query Methods for Retrieval and Reranking in Conversational Agents

TL;DR

The paper tackles conversational information-seeking in the iKAT track by extending the Generate-Retrieve-Generate pipeline with passage-query (PQ) generation to better align retrieval with target passages. It proposes two PQ-based variants, Weighted Reranking and Short and Long Passages, using a Meta Llama model to generate structured PQs and guide retrieval with BM25 and neural rerankers. Across experiments, PQ-based approaches outperform baselines and achieve performance comparable to GPT-4-based systems, with metrics such as $nDCG@3$, $P@20$, and $Recall@20$ indicating strong semantic alignment and multi-turn effectiveness while highlighting potential efficiency gains with smaller LLMs. The findings demonstrate that structured PQs can balance retrieval quality and computational cost, offering practical improvements for multi-turn conversational information seeking while outlining avenues for further PQ design optimization and prompting strategies.

Abstract

This paper presents our approach to the TREC Interactive Knowledge Assistance Track (iKAT), which focuses on improving conversational information-seeking (CIS) systems. While recent advancements in CIS have improved conversational agents' ability to assist users, significant challenges remain in understanding context and retrieving relevant documents across domains and dialogue turns. To address these issues, we extend the Generate-Retrieve-Generate pipeline by developing passage queries (PQs) that align with the target document's expected format to improve query-document matching during retrieval. We propose two variations of this approach: Weighted Reranking and Short and Long Passages. Each method leverages a Meta Llama model for context understanding and generating queries and responses. Passage ranking evaluation results show that the Short and Long Passages approach outperformed the organizers' baselines, performed best among Llama-based systems in the track, and achieved results comparable to GPT-4-based systems. These results indicate that the method effectively balances efficiency and performance. Findings suggest that PQs improve semantic alignment with target documents and demonstrate their potential to improve multi-turn dialogue systems.

Figures (3)

• Figure 1: System architecture
• Figure 2: Example PTKB statements
• Figure 3: Average nDCG@5 across topic turns