Optimal Dataset Size for Recommender Systems: Evaluating Algorithms' Performance via Downsampling
Ardalan Arabzadeh
TL;DR
This work investigates training-time data reduction in recommender systems as a pathway to greener AI. It evaluates seven datasets with two downsampling strategies (User-Based and User-Subset), two core pruning levels (10-core and 30-core), and 12 algorithms across LensKit, RecPack, and RecBole, using $nDCG@10$ as the primary performance metric. The study quantifies substantial runtime and $CO_2e$ emissions reductions (e.g., up to $82\%$ runtime reduction and $<\$~50 KgCO$_2$e per algorithm per dataset at moderate downsamples) while revealing that the impact on ranking quality is highly scenario-dependent, with some configurations maintaining or even surpassing full-dataset performance. Key findings show clear energy-efficiency gains, nuanced trade-offs across algorithms and datasets, and practical guidelines for deploying energy-aware recommender systems, contributing to the Green Recommender Systems literature. The results underscore that careful downsampling, combined with pruning choices and dataset characteristics, can yield sustainable and effective recommender solutions in real-world settings where resource constraints are critical.
Abstract
This thesis investigates dataset downsampling as a strategy to optimize energy efficiency in recommender systems while maintaining competitive performance. With increasing dataset sizes posing computational and environmental challenges, this study explores the trade-offs between energy efficiency and recommendation quality in Green Recommender Systems, which aim to reduce environmental impact. By applying two downsampling approaches to seven datasets, 12 algorithms, and two levels of core pruning, the research demonstrates significant reductions in runtime and carbon emissions. For example, a 30% downsampling portion can reduce runtime by 52% compared to the full dataset, leading to a carbon emission reduction of up to 51.02 KgCO2e during the training of a single algorithm on a single dataset. The analysis reveals that algorithm performance under different downsampling portions depends on factors like dataset characteristics, algorithm complexity, and the specific downsampling configuration (scenario dependent). Some algorithms, which showed lower nDCG@10 scores compared to higher-performing ones, exhibited lower sensitivity to the amount of training data, offering greater potential for efficiency in lower downsampling portions. On average, these algorithms retained 81% of full-size performance using only 50% of the training set. In certain downsampling configurations, where more users were progressively included while keeping the test set size fixed, they even showed higher nDCG@10 scores than when using the full dataset. These findings highlight the feasibility of balancing sustainability and effectiveness, providing insights for designing energy-efficient recommender systems and promoting sustainable AI practices.