Diffusion Recommender Models and the Illusion of Progress: A Concerning Study of Reproducibility and a Conceptual Mismatch

TL;DR

This study interrogates the reproducibility and conceptual fit of diffusion-based recommender models (DiffRec, CF-Diff, GiffCF, DDRM) reported at SIGIR 2023–2024. By porting the original artifacts to a common framework and re-evaluating with well-tuned baselines across multiple datasets, the authors reveal persistent methodological issues, substantial run-time variance, and frequent outperformance of the diffusion methods by simpler baselines. They argue that diffusion models, as applied in these works, often fail to deliver reliable gains for top-n recommendation and raise concerns about forward-process handling, limited data corruption, and over-constrained guidance. The findings advocate for greater reproducibility, rigorous hyperparameter tuning of baselines, and rethinking evaluation paradigms to ensure genuine progress. The work also discusses the substantial computational costs and highlights a broader need for changes in the research culture surrounding reproducibility and reporting in recommender systems research.

Abstract

Countless new machine learning models are published every year and are reported to significantly advance the state-of-the-art in \emph{top-n} recommendation. However, earlier reproducibility studies indicate that progress in this area may be quite limited. Specifically, various widespread methodological issues, e.g., comparisons with untuned baseline models, have led to an \emph{illusion of progress}. In this work, our goal is to examine whether these problems persist in today's research. To this end, we aim to reproduce the latest advancements reported from applying modern Denoising Diffusion Probabilistic Models to recommender systems, focusing on four models published at the top-ranked SIGIR conference in 2023 and 2024. Our findings are concerning, revealing persistent methodological problems. Alarmingly, through experiments, we find that the latest recommendation techniques based on diffusion models, despite their computational complexity and substantial carbon footprint, are consistently outperformed by simpler existing models. Furthermore, we identify key mismatches between the characteristics of diffusion models and those of the traditional \emph{top-n} recommendation task, raising doubts about their suitability for recommendation. We also note that, in the papers we analyze, the generative capabilities of these models are constrained to a minimum. Overall, our results and continued methodological issues call for greater scientific rigor and a disruptive change in the research and publication culture in this area.