CRS Arena: Crowdsourced Benchmarking of Conversational Recommender Systems
Nolwenn Bernard, Hideaki Joko, Faegheh Hasibi, Krisztian Balog
TL;DR
CRS Arena introduces a crowdsourced, pairwise benchmarking platform for Conversational Recommender Systems, enabling real-user battles between anonymous CRSs and collecting both outcomes and explicit satisfaction feedback. It demonstrates robustness across open and closed crowdsourcing settings and releases CRSArena-Dial, a dataset of 474 real-user conversations with an initial Elo-based ranking. The work highlights a disconnect between recall-focused metrics and user satisfaction, underscoring the value of holistic, interactive evaluation. By providing an open-source platform and a reusable CRS integration framework, the study lays the groundwork for scalable, realistic benchmarking and community-driven expansion of CRS evaluation resources.
Abstract
We introduce CRS Arena, a research platform for scalable benchmarking of Conversational Recommender Systems (CRS) based on human feedback. The platform displays pairwise battles between anonymous conversational recommender systems, where users interact with the systems one after the other before declaring either a winner or a draw. CRS Arena collects conversations and user feedback, providing a foundation for reliable evaluation and ranking of CRSs. We conduct experiments with CRS Arena on both open and closed crowdsourcing platforms, confirming that both setups produce highly correlated rankings of CRSs and conversations with similar characteristics. We release CRSArena-Dial, a dataset of 474 conversations and their corresponding user feedback, along with a preliminary ranking of the systems based on the Elo rating system. The platform is accessible at https://iai-group-crsarena.hf.space/.