Efficient Conversational Search via Topical Locality in Dense Retrieval

TL;DR

This work tackles the latency bottleneck in conversational search by exploiting topical locality across a conversation to prune the dense embedding search space. The authors introduce TopLoc, a server-side method that caches a subset of centroids for IVF and uses a privileged entry point for HNSW, integrating with two dense retrievers (Snowflake and Dragon) and FAISS. They provide formal definitions and proxy monitoring metrics (I and I0) to maintain retrieval quality while gaining substantial speedups, reporting up to 10.4x faster responses with minimal loss on TREC CAsT 2019/2020. The results demonstrate that TopLoc enables real-time, scalable conversational search with negligible degradation in effectiveness, and the work points to future improvements in topic-shift detection and dynamic cache refreshing.

Abstract

Pre-trained language models have been widely exploited to learn dense representations of documents and queries for information retrieval. While previous efforts have primarily focused on improving effectiveness and user satisfaction, response time remains a critical bottleneck of conversational search systems. To address this, we exploit the topical locality inherent in conversational queries, i.e., the tendency of queries within a conversation to focus on related topics. By leveraging query embedding similarities, we dynamically restrict the search space to semantically relevant document clusters, reducing computational complexity without compromising retrieval quality. We evaluate our approach on the TREC CAsT 2019 and 2020 datasets using multiple embedding models and vector indexes, achieving improvements in processing speed of up to 10.4X with little loss in performance (4.4X without any loss). Our results show that the proposed system effectively handles complex, multiturn queries with high precision and efficiency, offering a practical solution for real-time conversational search.

Figures (2)

• Figure 1: IVF, TopLoc$_{\text{IVF}}$, and TopLoc$_{\text{IVF+}}$ with Snowflake embeddings on TREC CAsT 2019 and 2020 by varying $np$.
• Figure 2: HNSW and TopLoc$_{\text{HNSW}}$ with Snowflake embeddings on TREC CAsT 2019 and 2020 by varying ef-search.