Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Incorporating External Knowledge and Goal Guidance for LLM-based Conversational Recommender Systems

Chuang Li, Yang Deng, Hengchang Hu, Min-Yen Kan, Haizhou Li

TL;DR

The paper tackles the challenge of applying LLMs to domain-specific conversational recommender systems by identifying the need for external knowledge and goal guidance. It introduces ChatCRS, a modular framework that integrates a knowledge retrieval agent and a goal-planning agent under an LLM-based conversational controller to ground responses in external KBs and proactively steer dialogues. Empirical results on DuRecDial and TG-Redial show state-of-the-art performance in response generation and a substantial, approximately tenfold, boost in recommendation accuracy when external inputs are leveraged, with human evaluations confirming improvements in informativeness and proactivity. The work demonstrates the feasibility and benefits of knowledge-grounded, goal-directed LLM-based CRS using few-shot in-context learning and parameter-efficient fine-tuning, highlighting practical significance for scalable, domain-adaptive conversational agents.

Abstract

This paper aims to efficiently enable large language models (LLMs) to use external knowledge and goal guidance in conversational recommender system (CRS) tasks. Advanced LLMs (e.g., ChatGPT) are limited in domain-specific CRS tasks for 1) generating grounded responses with recommendation-oriented knowledge, or 2) proactively leading the conversations through different dialogue goals. In this work, we first analyze those limitations through a comprehensive evaluation, showing the necessity of external knowledge and goal guidance which contribute significantly to the recommendation accuracy and language quality. In light of this finding, we propose a novel ChatCRS framework to decompose the complex CRS task into several sub-tasks through the implementation of 1) a knowledge retrieval agent using a tool-augmented approach to reason over external Knowledge Bases and 2) a goal-planning agent for dialogue goal prediction. Experimental results on two multi-goal CRS datasets reveal that ChatCRS sets new state-of-the-art benchmarks, improving language quality of informativeness by 17% and proactivity by 27%, and achieving a tenfold enhancement in recommendation accuracy.

Incorporating External Knowledge and Goal Guidance for LLM-based Conversational Recommender Systems

TL;DR

The paper tackles the challenge of applying LLMs to domain-specific conversational recommender systems by identifying the need for external knowledge and goal guidance. It introduces ChatCRS, a modular framework that integrates a knowledge retrieval agent and a goal-planning agent under an LLM-based conversational controller to ground responses in external KBs and proactively steer dialogues. Empirical results on DuRecDial and TG-Redial show state-of-the-art performance in response generation and a substantial, approximately tenfold, boost in recommendation accuracy when external inputs are leveraged, with human evaluations confirming improvements in informativeness and proactivity. The work demonstrates the feasibility and benefits of knowledge-grounded, goal-directed LLM-based CRS using few-shot in-context learning and parameter-efficient fine-tuning, highlighting practical significance for scalable, domain-adaptive conversational agents.

Abstract

This paper aims to efficiently enable large language models (LLMs) to use external knowledge and goal guidance in conversational recommender system (CRS) tasks. Advanced LLMs (e.g., ChatGPT) are limited in domain-specific CRS tasks for 1) generating grounded responses with recommendation-oriented knowledge, or 2) proactively leading the conversations through different dialogue goals. In this work, we first analyze those limitations through a comprehensive evaluation, showing the necessity of external knowledge and goal guidance which contribute significantly to the recommendation accuracy and language quality. In light of this finding, we propose a novel ChatCRS framework to decompose the complex CRS task into several sub-tasks through the implementation of 1) a knowledge retrieval agent using a tool-augmented approach to reason over external Knowledge Bases and 2) a goal-planning agent for dialogue goal prediction. Experimental results on two multi-goal CRS datasets reveal that ChatCRS sets new state-of-the-art benchmarks, improving language quality of informativeness by 17% and proactivity by 27%, and achieving a tenfold enhancement in recommendation accuracy.
Paper Structure (21 sections, 4 equations, 7 figures, 14 tables)

This paper contains 21 sections, 4 equations, 7 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary: Empirical Analysis
  4. Empirical Analysis Approaches
  5. Empirical Analysis Findings
  6. ChatCRS
  7. Knowledge Retrieval agent
  8. Goal Planning agent
  9. LLM-based Conversational Agent
  10. Experiments
  11. Experimental Setups
  12. Experimental Results
  13. Detailed Discussion
  14. Conclusion
  15. Appendix
  16. ...and 6 more sections

Figures (7)

  • Figure 1: An example of CRS tasks with external knowledge and goal guidance. (Blue: CRS tasks; Red: External Knowledge and Goal Guidance)
  • Figure 2: a) Empirical analysis of LLMs in CRS tasks with DG, COT& Oracle; b) System design of ChatCRS framework using LLMs as a conversational agent to control the goal planning and knowledge retrieval agents.
  • Figure 3: Knowledge retrieval agent in ChatCRS.
  • Figure 4: Knowledge ratio for each goal type on DuRecDial. (X-axis: Knowledge Ratio ; Y-axis: Goal type)
  • Figure 5: ICL prompt design for empirical analysis, detailed examples are shown in Appendix \ref{['Prompt']}.
  • ...and 2 more figures