Selecting Query-bag as Pseudo Relevance Feedback for Information-seeking Conversations

TL;DR

This paper addresses the limitation of relying solely on the current user query in information-seeking conversations by introducing a Query-bag Pseudo Relevance Feedback framework (QB-PRF). It combines a Query-bag Selection module (QBS) that uses contrastive learning with a Variational Auto-Encoder to assemble a diverse set of synonymous queries, and a Query-bag Fusion module (QBF) that fuses this bag with the original query using a two-layer transformer with cross- and self-attention. The approach is validated on two backbones, BERT and GPT-2, across LaiYe and Quora datasets, showing improved retrieval/matching performance over strong baselines, with ablations confirming the importance of both QBS and QBF. The results demonstrate the practical potential of query-bag based pseudo relevance signals to refine query representations and boost information-seeking conversation quality, though limitations such as scarce query-bag candidates and potential extension to other tasks are acknowledged.

Abstract

Information-seeking dialogue systems are widely used in e-commerce systems, with answers that must be tailored to fit the specific settings of the online system. Given the user query, the information-seeking dialogue systems first retrieve a subset of response candidates, then further select the best response from the candidate set through re-ranking. Current methods mainly retrieve response candidates based solely on the current query, however, incorporating similar questions could introduce more diverse content, potentially refining the representation and improving the matching process. Hence, in this paper, we proposed a Query-bag based Pseudo Relevance Feedback framework (QB-PRF), which constructs a query-bag with related queries to serve as pseudo signals to guide information-seeking conversations. Concretely, we first propose a Query-bag Selection module (QBS), which utilizes contrastive learning to train the selection of synonymous queries in an unsupervised manner by leveraging the representations learned from pre-trained VAE. Secondly, we come up with a Query-bag Fusion module (QBF) that fuses synonymous queries to enhance the semantic representation of the original query through multidimensional attention computation. We verify the effectiveness of the QB-PRF framework on two competitive pretrained backbone models, including BERT and GPT-2. Experimental results on two benchmark datasets show that our framework achieves superior performance over strong baselines.

Figures (3)

• Figure 1: The comparison of existing baseline and our framework for information-seeking conversation. We use query-bag to enhance the query representation so as to enhance re-ranking performance.
• Figure 2: The structure of the QB-PRF framework. (a) QBS module, employs the $\mathcal{L}_{reward}$ and $\mathcal{L}_{b}$ to supervise the selection of similar queries. (b) QBF module, utilizes the multidimensional attention for a refined query representation.
• Figure 3: (a) Recall denotes the size of query-bag in datasets with regard to different candidate set size after dense retrieval in Figure \ref{['fig:qb_structure']}. (b) Accuracy refers to the proportion that belong to the query-bag with different candidate set size. (c) Performance on different evaluation steps. We evaluate the performance every 0.1 epoch.