Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestion

Jinheon Baek, Nirupama Chandrasekaran, Silviu Cucerzan, Allen herring, Sujay Kumar Jauhar

TL;DR

This work introduces K-LaMP, a lightweight, privacy-conscious framework that personalizes LLM outputs by augmenting prompts with an entity-centric knowledge store built from users' search histories. By grounding contextual query suggestions in both user interests and domain knowledge, K-LaMP outperforms several baselines in human evaluations on Bing log data, with retrieval from the entity store proving especially effective. The approach leverages existing search logging infrastructure and modern entity linking to achieve scalable personalization without full user profiling, and the authors discuss extensions to other search tasks and domains. Overall, the paper demonstrates the viability and benefits of knowledge-grounded personalization for LLMs in real-world, web-scale settings.

Abstract

Large Language Models (LLMs) excel at tackling various natural language tasks. However, due to the significant costs involved in re-training or fine-tuning them, they remain largely static and difficult to personalize. Nevertheless, a variety of applications could benefit from generations that are tailored to users' preferences, goals, and knowledge. Among them is web search, where knowing what a user is trying to accomplish, what they care about, and what they know can lead to improved search experiences. In this work, we propose a novel and general approach that augments an LLM with relevant context from users' interaction histories with a search engine in order to personalize its outputs. Specifically, we construct an entity-centric knowledge store for each user based on their search and browsing activities on the web, which is then leveraged to provide contextually relevant LLM prompt augmentations. This knowledge store is light-weight, since it only produces user-specific aggregate projections of interests and knowledge onto public knowledge graphs, and leverages existing search log infrastructure, thereby mitigating the privacy, compliance, and scalability concerns associated with building deep user profiles for personalization. We validate our approach on the task of contextual query suggestion, which requires understanding not only the user's current search context but also what they historically know and care about. Through a number of experiments based on human evaluation, we show that our approach is significantly better than several other LLM-powered baselines, generating query suggestions that are contextually more relevant, personalized, and useful.

Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestion

TL;DR

This work introduces K-LaMP, a lightweight, privacy-conscious framework that personalizes LLM outputs by augmenting prompts with an entity-centric knowledge store built from users' search histories. By grounding contextual query suggestions in both user interests and domain knowledge, K-LaMP outperforms several baselines in human evaluations on Bing log data, with retrieval from the entity store proving especially effective. The approach leverages existing search logging infrastructure and modern entity linking to achieve scalable personalization without full user profiling, and the authors discuss extensions to other search tasks and domains. Overall, the paper demonstrates the viability and benefits of knowledge-grounded personalization for LLMs in real-world, web-scale settings.

Abstract

Large Language Models (LLMs) excel at tackling various natural language tasks. However, due to the significant costs involved in re-training or fine-tuning them, they remain largely static and difficult to personalize. Nevertheless, a variety of applications could benefit from generations that are tailored to users' preferences, goals, and knowledge. Among them is web search, where knowing what a user is trying to accomplish, what they care about, and what they know can lead to improved search experiences. In this work, we propose a novel and general approach that augments an LLM with relevant context from users' interaction histories with a search engine in order to personalize its outputs. Specifically, we construct an entity-centric knowledge store for each user based on their search and browsing activities on the web, which is then leveraged to provide contextually relevant LLM prompt augmentations. This knowledge store is light-weight, since it only produces user-specific aggregate projections of interests and knowledge onto public knowledge graphs, and leverages existing search log infrastructure, thereby mitigating the privacy, compliance, and scalability concerns associated with building deep user profiles for personalization. We validate our approach on the task of contextual query suggestion, which requires understanding not only the user's current search context but also what they historically know and care about. Through a number of experiments based on human evaluation, we show that our approach is significantly better than several other LLM-powered baselines, generating query suggestions that are contextually more relevant, personalized, and useful.
Paper Structure (17 sections, 4 figures, 11 tables)

This paper contains 17 sections, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. K-LaMP: Knowledge-Augmented LLMs for Personalized Query Suggestion
  4. Problem Statement
  5. Knowledge Store for Output Personalization
  6. Constructing Personal Knowledge Stores
  7. Contextual Retrieval from Personal Knowledge Stores
  8. Experimental Setup
  9. Data
  10. Baselines and Our Model
  11. Evaluation Setup
  12. Experimental Results
  13. Additional Studies and Analyses
  14. Automatic Evaluation Setup
  15. Conclusion
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Illustration of our proposed Knowledge-augmented large Language Model for Personalization (K-LaMP) framework for contextual query suggestion. (A) A user's search context includes a current query and an page being viewed. (B) The user has knowledge of medicine, extracted from past search activities. (C) Conventional query suggestion with na√Øve LLMs generates a query unrelated to the user's knowledge and the context. (D) K-LaMP suggests a query that is personally and contextually relevant.
  • Figure 2: Overview of K-LaMP. The inputs to the system are (A) previous search logs and (B) the current search context of a user, which consist of a query and an associated web-page. (C) Each search record is stored in a memory stream along with a time stamp. (D) The entity-based knowledge store is constructed by aggregating entities extracted from the memory stream. (E) K-LaMP augmented with entities retrieved from the entity-based personal knowledge store, generates query suggestions that are compatible with the user's knowledge and interests.
  • Figure 3: Results of variants of K-LaMP on personalized knowledge retrieval strategy, and results without retrieval.
  • Figure 4: Results of baselines and K-LaMP with different numbers of previous queries (i.e., different memory sizes).