De-centering the (Traditional) User: Multistakeholder Evaluation of Recommender Systems

TL;DR

This paper advocates for multistakeholder evaluation of recommender systems, arguing that end-user metrics alone fail to capture the wider ecosystem impacts. It lays out a value-centered methodology—identifying stakeholders, elicit ing values and goals, translating them into metrics, and aggregating results—and demonstrates it through three domain exemplars (music streaming, education, HR). By detailing stakeholder-specific metrics, aggregation strategies, transparency considerations, and governance, the work provides practical guidance for implementing holistic, domain-sensitive evaluations while acknowledging trade-offs and the need for iterative, participatory processes. The contribution is a structured framework that enables researchers and practitioners to assess and govern recommender systems across multiple stakeholders, with implications for policy, design, and accountability in real-world deployments.

Abstract

Multistakeholder recommender systems are those that account for the impacts and preferences of multiple groups of individuals, not just the end users receiving recommendations. Due to their complexity, these systems cannot be evaluated strictly by the overall utility of a single stakeholder, as is often the case of more mainstream recommender system applications. In this article, we focus our discussion on the challenges of multistakeholder evaluation of recommender systems. We bring attention to the different aspects involved -- from the range of stakeholders involved (including but not limited to providers and consumers) to the values and specific goals of each relevant stakeholder. We discuss how to move from theoretical principles to practical implementation, providing specific use case examples. Finally, we outline open research directions for the RecSys community to explore. We aim to provide guidance to researchers and practitioners about incorporating these complex and domain-dependent issues of evaluation in the course of designing, developing, and researching applications with multistakeholder aspects.

Figures (5)

• Figure 1: A multistakeholder view of a recommendation ecosystem
• Figure 2: Stakeholder configuration for the music streaming example.
• Figure 3: Recommendation relevance vs. profit (adapted from abdollahpouri2020multistakeholderjannach2017vams)
• Figure 4: Stakeholder configuration for the education example
• Figure 5: Stakeholder configuration for the human resources example