Stop Playing the Guessing Game! Target-free User Simulation for Evaluating Conversational Recommender Systems

TL;DR

This work tackles the challenge of evaluating Conversational Recommender Systems beyond simplistic target-guessing by introducing PEPPER, a protocol that uses target-free user simulators grounded in real user histories and reviews to simulate natural preference elicitation. It defines new quantitative and qualitative metrics, notably Preference Coverage (PC) and Preference Coverage Increase Rate (PCIR), as well as proactiveness, coherence, and personalization scores evaluated via an LLM, to holistically assess a CRS’s ability to elicit preferences and provide accurate recommendations. Extensive experiments show that target-free simulators reduce bias, improve diversity of user interactions, and yield more reliable assessments of elicitation capability across diverse baselines and domains, including movie and e-commerce datasets. The findings underscore the limitations of Recall@$K$ as a multi-turn evaluation metric and demonstrate that PEPPER offers a robust framework for evaluating and comparing CRSs in more realistic conversational settings, with potential generalization to open-source models and cross-domain applications.

Abstract

Recent approaches in Conversational Recommender Systems (CRSs) have tried to simulate real-world users engaging in conversations with CRSs to create more realistic testing environments that reflect the complexity of human-agent dialogue. Despite the significant advancements, reliably evaluating the capability of CRSs to elicit user preferences still faces a significant challenge. Existing evaluation metrics often rely on target-biased user simulators that assume users have predefined preferences, leading to interactions that devolve into simplistic guessing game. These simulators typically guide the CRS toward specific target items based on fixed attributes, limiting the dynamic exploration of user preferences and struggling to capture the evolving nature of real-user interactions. Additionally, current evaluation metrics are predominantly focused on single-turn recall of target items, neglecting the intermediate processes of preference elicitation. To address this, we introduce PEPPER, a novel CRS evaluation protocol with target-free user simulators constructed from real-user interaction histories and reviews. PEPPER enables realistic user-CRS dialogues without falling into simplistic guessing games, allowing users to gradually discover their preferences through enriched interactions, thereby providing a more accurate and reliable assessment of the CRS's ability to elicit personal preferences. Furthermore, PEPPER presents detailed measures for comprehensively evaluating the preference elicitation capabilities of CRSs, encompassing both quantitative and qualitative measures that capture four distinct aspects of the preference elicitation process. Through extensive experiments, we demonstrate the validity of PEPPER as a simulation environment and conduct a thorough analysis of how effectively existing CRSs perform in preference elicitation and recommendation.

Figures (7)

• Figure 1: While existing target-biased user simulators directly reveal attributes of target items for CRS evaluation (Upper), our target-free user simulator engages with more general preference (Lower), making preference elicitation crucial to provide accurate recommendations.
• Figure 2: Comparison between selected and residual recall for revealing target-item reliance in user simulators.
• Figure 3: (Upper) Recall@50 of the different CRSs across 20 dialogue turns on the IMDBReDial dataset. (Lower) Average Jaccard distance between consecutive recommendation lists of CRS at each turn.
• Figure 4: Overview of PEPPER. Within our protocol, a user simulator and a CRS interact via (1) item interface and (2) dialogue interface. The user simulator is initialized with general preferences derived from real-world datasets (i.e., IMDB). [Blue line] At each interaction, the user simulator first inspects top-$k$ recommendations in the item interface, classifying the items into seen and unseen sets. It then uses these classifications and the general preferences to generate reflected preferences. Finally, it provides a tailored response enriched with detailed personal preferences. [Green line] In response, the CRS generates an utterance and presents new item recommendations.
• Figure 5: Human evaluation on the quality of generated dialogues from Target-free vs Target-biased simulator.