Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fuzzy synthetic method for evaluating explanations in recommender systems

Jinfeng Zhong, Elsa Negre

TL;DR

The paper addresses the challenge of evaluating explanations in recommender systems across multiple user-centered goals. It combines a web-based interactive questionnaire with a context-aware recommendation method and introduces a fuzzy synthetic evaluation to aggregate six metrics: efficiency, effectiveness, persuasiveness, trust, transparency, and satisfaction. Through a user study on the CoMoDa dataset, it compares context-free and context-aware explanations, revealing that context-aware explanations increase decision time and yield mixed objective outcomes, while subjective perceptions can diverge from measured effects. The fuzzy aggregation provides a principled, uncertainty-aware overall assessment, offering a replicable evaluation framework to guide the design of context-aware explanations in RSs and to balance trade-offs among competing goals in explanations.

Abstract

Recommender systems aim to help users find relevant items more quickly by providing personalized recommendations. Explanations in recommender systems help users understand why such recommendations have been generated, which in turn makes the system more transparent and promotes users' trust and satisfaction. In recent years, explaining recommendations has drawn increasing attention from both academia and from industry. In this paper, we present a user study to investigate context-aware explanations in recommender systems. In particular, we build a web-based questionnaire that is able to interact with users: generating and explaining recommendations. With this questionnaire, we investigate the effects of context-aware explanations in terms of efficiency, effectiveness, persuasiveness, satisfaction, trust and transparency through a user study. Besides, we propose a novel method based on fuzzy synthetic evaluation for aggregating these metrics.

Fuzzy synthetic method for evaluating explanations in recommender systems

TL;DR

The paper addresses the challenge of evaluating explanations in recommender systems across multiple user-centered goals. It combines a web-based interactive questionnaire with a context-aware recommendation method and introduces a fuzzy synthetic evaluation to aggregate six metrics: efficiency, effectiveness, persuasiveness, trust, transparency, and satisfaction. Through a user study on the CoMoDa dataset, it compares context-free and context-aware explanations, revealing that context-aware explanations increase decision time and yield mixed objective outcomes, while subjective perceptions can diverge from measured effects. The fuzzy aggregation provides a principled, uncertainty-aware overall assessment, offering a replicable evaluation framework to guide the design of context-aware explanations in RSs and to balance trade-offs among competing goals in explanations.

Abstract

Recommender systems aim to help users find relevant items more quickly by providing personalized recommendations. Explanations in recommender systems help users understand why such recommendations have been generated, which in turn makes the system more transparent and promotes users' trust and satisfaction. In recent years, explaining recommendations has drawn increasing attention from both academia and from industry. In this paper, we present a user study to investigate context-aware explanations in recommender systems. In particular, we build a web-based questionnaire that is able to interact with users: generating and explaining recommendations. With this questionnaire, we investigate the effects of context-aware explanations in terms of efficiency, effectiveness, persuasiveness, satisfaction, trust and transparency through a user study. Besides, we propose a novel method based on fuzzy synthetic evaluation for aggregating these metrics.
Paper Structure (15 sections, 2 figures, 6 tables)

This paper contains 15 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Explanations in recommender systems
  4. Metrics
  5. Our proposition
  6. Recommendation method
  7. Experiment setup
  8. Baselines
  9. Results and discussion
  10. Objective evaluation
  11. Subjective evaluation
  12. Correlation between different metrics
  13. Aggregating metrics
  14. Discussions
  15. Conclusion and perspectives

Figures (2)

  • Figure 1: The number indicates the movie recommended. White indicates that only explanation is displayed, black indicates that detailed information of the movie is displayed, $r$ is the rating given by participants, and $t$ is the time cost by participants to give a rating.
  • Figure 2: A screenshot of the Likert-scale questionnaire when evaluating the effectiveness of explanations. Participants are indicated that 1 means they think the explanation does not help them make good decision at all while 5 indicates that the explanation help them make good decisions.