Evaluation on Entity Matching in Recommender Systems
Zihan Huang, Rohan Surana, Zhouhang Xie, Junda Wu, Yu Xia, Julian McAuley
TL;DR
This work tackles the lack of rigorous evaluation frameworks for cross-dataset entity matching in recommender systems by introducing Reddit-Amazon-EM, a manually annotated dataset linking Reddit movie mentions to Amazon catalog entries. The authors benchmark a spectrum of EM methods—including graph-based GNEM, LLM-enhanced ComEM, and hybrid Embedding+Fuzzy approaches—demonstrating that graph-based and LLMed methods outperform traditional lexical and embedding baselines on both EM and downstream CRS tasks. GNEM achieves the strongest EM performance (F1 ≈ 96.3%, Accuracy ≈ 96.7%), with ComEM close behind, while the CRS case study reveals that EM grounding improves conversational recommendations, though gains are moderated by LLM quality. By releasing the dataset, annotations, and evaluation code, the study provides a reproducible benchmark to advance entity matching research for knowledge-grounded and conversational recommender systems.
Abstract
Entity matching is a crucial component in various recommender systems, including conversational recommender systems (CRS) and knowledge-based recommender systems. However, the lack of rigorous evaluation frameworks for cross-dataset entity matching impedes progress in areas such as LLM-driven conversational recommendations and knowledge-grounded dataset construction. In this paper, we introduce Reddit-Amazon-EM, a novel dataset comprising naturally occurring items from Reddit and the Amazon '23 dataset. Through careful manual annotation, we identify corresponding movies across Reddit-Movies and Amazon'23, two existing recommender system datasets with inherently overlapping catalogs. Leveraging Reddit-Amazon-EM, we conduct a comprehensive evaluation of state-of-the-art entity matching methods, including rule-based, graph-based, lexical-based, embedding-based, and LLM-based approaches. For reproducible research, we release our manually annotated entity matching gold set and provide the mapping between the two datasets using the best-performing method from our experiments. This serves as a valuable resource for advancing future work on entity matching in recommender systems.