Informed Dataset Selection
Abdullah Abbas, Michael Heep, Theodor Sperle
TL;DR
The paper tackles the lack of systematic dataset selection in recommender systems by introducing the APS Explorer, a web tool that implements the Algorithm Performance Space (APS) framework. It analyzes 96 datasets with 28 algorithms across three metrics (nDCG, HR, Recall) at five K-values, and extends APS with a quintile-based dataset difficulty classification and a variance-normalized Mahalanobis-distance-based similarity measure, transformed via an exponential decay to a 0–1 confidence. The tool provides three interactive modules—performance visualization (APS), direct algorithm comparison, and dataset metadata—facilitating evidence-based, diverse, and reproducible dataset selection. By making these capabilities publicly available, the APS Explorer aims to improve robustness and generalizability in benchmarking recommender systems and guiding dataset choice beyond popularity or familiarity.
Abstract
The selection of datasets in recommender systems research lacks a systematic methodology. Researchers often select datasets based on popularity rather than empirical suitability. We developed the APS Explorer, a web application that implements the Algorithm Performance Space (APS) framework for informed dataset selection. The system analyzes 96 datasets using 28 algorithms across three metrics (nDCG, Hit Ratio, Recall) at five K-values. We extend the APS framework with a statistical based classification system that categorizes datasets into five difficulty levels based on quintiles. We also introduce a variance-normalized distance metric based on Mahalanobis distance to measure similarity. The APS Explorer was successfully developed with three interactive modules for visualizing algorithm performance, direct comparing algorithms, and analyzing dataset metadata. This tool shifts the process of selecting datasets from intuition-based to evidence-based practices, and it is publicly available at datasets.recommender-systems.com.