Algorithm Performance Spaces for Strategic Dataset Selection

TL;DR

This work tackles reproducibility and meaningfulness in offline recommender evaluation by introducing the Algorithm Performance Space (APS), a framework that differentiates datasets based on measured algorithm performance. It defines three metrics—DifficultyAPS, VarianceAPS, and DiversityAPS—and uses visualization tools like Mini-APS and PCA to analyze high-dimensional performance data. An empirical study with 75 datasets and 5 algorithms demonstrates that APS can reveal dataset diversity and dataset–algorithm interaction patterns, highlighting clustering of MovieLens data and variability among Amazon datasets. The results validate the APS concept and highlight future directions for broader adoption, richer datasets, and more robust evaluation setups to support principled dataset selection in recommender system research.

Abstract

The evaluation of new algorithms in recommender systems frequently depends on publicly available datasets, such as those from MovieLens or Amazon. Some of these datasets are being disproportionately utilized primarily due to their historical popularity as baselines rather than their suitability for specific research contexts. This thesis addresses this issue by introducing the Algorithm Performance Space, a novel framework designed to differentiate datasets based on the measured performance of algorithms applied to them. An experimental study proposes three metrics to quantify and justify dataset selection to evaluate new algorithms. These metrics also validate assumptions about datasets, such as the similarity between MovieLens datasets of varying sizes. By creating an Algorithm Performance Space and using the proposed metrics, differentiating datasets was made possible, and diverse dataset selections could be found. While the results demonstrate the framework's potential, further research proposals and implications are discussed to develop Algorithm Performance Spaces tailored to diverse use cases.

Figures (8)

• Figure 1: Illustration of the Algorithm Performance Space (APS) as introduced by beel2024informed. The two axes represent the measured performance of algorithms A1 and A2. The orange points 1 to 16 represent datasets. These datasets are clustered based on similar performance on algorithms A1 and A2 and can now be differentiated.
• Figure 4: Four mini-APS as representation for the full APS. MovieLens datasets are highlighted by violet markers, Amazon datasets by black ones. Axes denote normalized nDCG@10 from 0 to 1 (best-performing nDCG of $\approx$ 0.5).
• Figure 5: The same four mini-APS chosen in Figure \ref{['fig:miniAPSnew']} from beel2024informed for comparison. MovieLens datasets are highlighted by violet markers, Amazon datasets by black ones. Axes denote normalized nDCG@10 from 0 to 1 (best-performing nDCG of $\approx$ 0.5).
• Figure 6: Results of dimensional reduction via PCA to two dimensions. Axes show both calculated components with explained variance as percentage. MovieLens datasets are highlighted by violet markers, Amazon datasets by black ones.
• Figure 7: The results from the PCA shown in Figure \ref{['fig:PCAcompare']} on the left, color-coded with DifficultyAPS calculated for each dataset.