Beyond Retrieval: Generating Narratives in Conversational Recommender Systems

TL;DR

This work tackles the challenge of generating natural, personalized narratives in conversational recommender systems by bridging recommender signals with large language models. It introduces REGEN, a narratives-enhanced extension of Amazon Reviews, and a fusion architecture that combines collaborative-filtering embeddings with semantic content embeddings as input to an LLM. Empirical results show that CF+Content fusion improves language-generation metrics by up to 12% in context-rich tasks, with ablation and embedding analyses providing insights into how signals are integrated. The dataset and baseline architecture advance reproducibility and provide a foundation for future exploration of narrative generation in recommendation settings.

Abstract

The recent advances in Large Language Model's generation and reasoning capabilities present an opportunity to develop truly conversational recommendation systems. However, effectively integrating recommender system knowledge into LLMs for natural language generation which is tailored towards recommendation tasks remains a challenge. This paper addresses this challenge by making two key contributions. First, we introduce a new dataset (REGEN) for natural language generation tasks in conversational recommendations. REGEN (Reviews Enhanced with GEnerative Narratives) extends the Amazon Product Reviews dataset with rich user narratives, including personalized explanations of product preferences, product endorsements for recommended items, and summaries of user purchase history. REGEN is made publicly available to facilitate further research. Furthermore, we establish benchmarks using well-known generative metrics, and perform an automated evaluation of the new dataset using a rater LLM. Second, the paper introduces a fusion architecture (CF model with an LLM) which serves as a baseline for REGEN. And to the best of our knowledge, represents the first attempt to analyze the capabilities of LLMs in understanding recommender signals and generating rich narratives. We demonstrate that LLMs can effectively learn from simple fusion architectures utilizing interaction-based CF embeddings, and this can be further enhanced using the metadata and personalization data associated with items. Our experiments show that combining CF and content embeddings leads to improvements of 4-12% in key language metrics compared to using either type of embedding individually. We also provide an analysis to interpret how CF and content embeddings contribute to this new generative task.

Figures (8)

• Figure 1: Purchase reason by rated attributes.
• Figure 2: User summary by rated attributes.
• Figure 3: User richness scores and associated confidence.
• Figure 4: Overview of the Model Architecture.
• Figure 5: Comparing Soft prompt embeddings vs LLM token embeddings. Soft prompt embeddings generated for all items in Office Products.