Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Conversational Recommendation with Contextual Adaptation of External Recommenders and LLM-based Reranking

Chuang Li, Weida Liang, Hengchang Hu, See-Kiong Ng, Min-Yen Kan, Haizhou Li, Yang Deng

Abstract

We tackle the challenge of integrating large language models (LLMs) with external recommender systems to enhance domain expertise in conversational recommendation (CRS). Current LLM-based CRS approaches primarily rely on zero/few-shot methods for generating item recommendations based on user queries, but this method faces two significant challenges: (1) without domain-specific adaptation, LLMs frequently recommend items not in the target item space, resulting in low recommendation accuracy; and (2) LLMs largely rely on dialogue context for content-based recommendations, neglecting the collaborative relationships among item sequences. To address these limitations, we introduce the CARE (Contextual Adaptation of Recommenders) framework. CARE (a) integrates external recommender systems as domain experts, producing candidate items through entity-level insights, and (b) customizes LLMs as rerankers to enhance the accuracy by leveraging contextual information. Our results demonstrate that incorporating CARE framework significantly enhances recommendation accuracy of LLMs by an average of 54% and 25% for ReDial and INSPIRED datasets. The most effective CARE strategy involves LLMs selecting and reranking candidate items that external recommenders provide based on contextual insights.

Improving Conversational Recommendation with Contextual Adaptation of External Recommenders and LLM-based Reranking

Abstract

We tackle the challenge of integrating large language models (LLMs) with external recommender systems to enhance domain expertise in conversational recommendation (CRS). Current LLM-based CRS approaches primarily rely on zero/few-shot methods for generating item recommendations based on user queries, but this method faces two significant challenges: (1) without domain-specific adaptation, LLMs frequently recommend items not in the target item space, resulting in low recommendation accuracy; and (2) LLMs largely rely on dialogue context for content-based recommendations, neglecting the collaborative relationships among item sequences. To address these limitations, we introduce the CARE (Contextual Adaptation of Recommenders) framework. CARE (a) integrates external recommender systems as domain experts, producing candidate items through entity-level insights, and (b) customizes LLMs as rerankers to enhance the accuracy by leveraging contextual information. Our results demonstrate that incorporating CARE framework significantly enhances recommendation accuracy of LLMs by an average of 54% and 25% for ReDial and INSPIRED datasets. The most effective CARE strategy involves LLMs selecting and reranking candidate items that external recommenders provide based on contextual insights.

Paper Structure

This paper contains 26 sections, 5 equations, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Main Problems and Key Challenges
  5. Item Space Discrepancy.
  6. Item-specific Information Negligence.
  7. CARE-CRS
  8. Transformer-based Recommender for Sequential Modelling
  9. Prompting LLMs for Contextual Adaptation
  10. Adaptation of Recommender as Domain Expert.
  11. Contextual Engagement for Conversational Recommendations.
  12. Experiments
  13. Setup
  14. Experimental Results (RQ1)
  15. Recommendation Performance.
  16. ...and 11 more sections

Figures (9)

  • Figure 1: CRS input example (ReDial Dataset) with conversational, contextual and entity-level inputs. (Blue: item entities; Red: attribute entities)
  • Figure 2: Item space discrepancy between LLM and target domain. (Blue Dot: good recommendation results; Red Dot: recommendations outside target domain; White Cross: items out of LLM's internal knowledge)
  • Figure 3: Ablation study for LLM-based CRS with different levels (conversational or entity) inputs. (Yellow: original conversational inputs in datasets; Orange: Contextual inputs w/o items; Gray: Entity-level inputs w/o conversations)
  • Figure 4: CARE Framework: Conversational inputs are firstly passed to an external recommender for entity-level sequential modelling (§ \ref{['seq_model']}). The candidate items output and original dialogue history are jointly used to prompt LLMs for the contextual adaptation and generate conversational recommendations (§ \ref{['prompt_llm']}).
  • Figure 5: Comparison between open-sourced LLMs and ChatGPT in recommendation accuracy.
  • ...and 4 more figures