SciNUP: Natural Language User Interest Profiles for Scientific Literature Recommendation

TL;DR

SciNUP addresses the lack of public, large-scale NL-profile datasets for scholarly recommendation by introducing a synthetic, arXiv-based benchmark that generates NL user profiles from authors' early publications and derives ground-truth items from later references. It evaluates a spectrum of retrieval and reranking approaches—from BM25 RM3 to cross-encoders and LLM-driven PRP reranking—plus an ensemble, revealing that while advanced methods close gaps, simple sparse retrieval remains competitive and the signals are complementary. The study shows that NL-profile breadth affects performance and that ensemble fusion yields the strongest results, underscoring the potential for hybrid models that integrate diverse signals. By making SciNUP publicly available, the work provides a reproducible platform to advance NL-profile-based scholarly recommendation and motivates further exploration of NL-profile generation strategies and human-centric validation.

Abstract

The use of natural language (NL) user profiles in recommender systems offers greater transparency and user control compared to traditional representations. However, there is scarcity of large-scale, publicly available test collections for evaluating NL profile-based recommendation. To address this gap, we introduce SciNUP, a novel synthetic dataset for scholarly recommendation that leverages authors' publication histories to generate NL profiles and corresponding ground truth items. We use this dataset to conduct a comparison of baseline methods, ranging from sparse and dense retrieval approaches to state-of-the-art LLM-based rerankers. Our results show that while baseline methods achieve comparable performance, they often retrieve different items, indicating complementary behaviors. At the same time, considerable headroom for improvement remains, highlighting the need for effective NL-based recommendation approaches. The SciNUP dataset thus serves as a valuable resource for fostering future research and development in this area.

Figures (7)

• Figure 1: Example NL user interest profile from our dataset.
• Figure 2: Overview of our approach.
• Figure 3: Prompt template used for NL profile generation. Placeholders for variables are in [square brackets], while conditional logic is marked by $\langle$IF$\rangle$...$\langle$/IF$\rangle$.
• Figure 4: Prompt template used for NL profile classification. Placeholders for variables are in [square brackets].
• Figure 5: Category and profile distributions in the SciNUP dataset.