How to Leverage Personal Textual Knowledge for Personalized Conversational Information Retrieval

TL;DR

This work tackles personalized CIR by leveraging a user’s PTKB to reformulate queries, addressing the challenge that PTKB is often noisy. It systematically compares three PTKB annotation approaches (Human, Automatic, LLM) and introduces two LLM-driven reformulation strategies, STR and SAR, enhanced by in-context learning. The key finding is that automatic PTKB annotation aligned with retrieval impact typically yields better results than human judgments, while LLM-based reformulation with targeted guidance can improve personalized retrieval, especially under few-shot prompts. Overall, the study highlights the potential and caveats of using PTKB for personalization and points to selective, data-efficient approaches as a promising direction for future CIR systems.

Abstract

Personalized conversational information retrieval (CIR) combines conversational and personalizable elements to satisfy various users' complex information needs through multi-turn interaction based on their backgrounds. The key promise is that the personal textual knowledge base (PTKB) can improve the CIR effectiveness because the retrieval results can be more related to the user's background. However, PTKB is noisy: not every piece of knowledge in PTKB is relevant to the specific query at hand. In this paper, we explore and test several ways to select knowledge from PTKB and use it for query reformulation by using a large language model (LLM). The experimental results show the PTKB might not always improve the search results when used alone, but LLM can help generate a more appropriate personalized query when high-quality guidance is provided.

Figures (2)

• Figure 1: The statistic of three different types of PTKB annotation. The "overlap" denotes the same annotation as Human.
• Figure 2: The results of three types of annotation approach with different reformulated query forms and retrievers.