Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

With Friends Like These, Who Needs Explanations? Evaluating User Understanding of Group Recommendations

Cedric Waterschoot, Raciel Yera Toledo, Nava Tintarev, Francesco Barile

TL;DR

This study investigates whether textual or multimodal explanations improve user understanding of group recommendations produced by social-choice-based aggregation. In a preregistered randomized trial (n = 271) with 3 aggregation strategies (ADD, LMS, APP) and 3 explanation conditions (no, textual, multimodal), the authors measure objective (model simulation, counterfactuals, error detection) and subjective understanding. The findings show explanations do not significantly affect understanding, while aggregation strategy significantly influences understandability, with LMS improving objective understanding and ADD enhancing subjective understanding; counterfactual tasks are particularly challenging. The work suggests that design focus should shift from explanations to the choice of aggregation mechanism and that evaluating understanding benefits from diverse tasks rather than a single metric, with practical implications for developing transparent, user-friendly GRS.

Abstract

Group Recommender Systems (GRS) employing social choice-based aggregation strategies have previously been explored in terms of perceived consensus, fairness, and satisfaction. At the same time, the impact of textual explanations has been examined, but the results suggest a low effectiveness of these explanations. However, user understanding remains fairly unexplored, even if it can contribute positively to transparent GRS. This is particularly interesting to study in more complex or potentially unfair scenarios when user preferences diverge, such as in a minority scenario (where group members have similar preferences, except for a single member in a minority position). In this paper, we analyzed the impact of different types of explanations on user understanding of group recommendations. We present a randomized controlled trial (n = 271) using two between-subject factors: (i) the aggregation strategy (additive, least misery, and approval voting), and (ii) the modality of explanation (no explanation, textual explanation, or multimodal explanation). We measured both subjective (self-perceived by the user) and objective understanding (performance on model simulation, counterfactuals and error detection). In line with recent findings on explanations for machine learning models, our results indicate that more detailed explanations, whether textual or multimodal, did not increase subjective or objective understanding. However, we did find a significant effect of aggregation strategies on both subjective and objective understanding. These results imply that when constructing GRS, practitioners need to consider that the choice of aggregation strategy can influence the understanding of users. Post-hoc analysis also suggests that there is value in analyzing performance on different tasks, rather than through a single aggregated metric of understanding.

With Friends Like These, Who Needs Explanations? Evaluating User Understanding of Group Recommendations

TL;DR

This study investigates whether textual or multimodal explanations improve user understanding of group recommendations produced by social-choice-based aggregation. In a preregistered randomized trial (n = 271) with 3 aggregation strategies (ADD, LMS, APP) and 3 explanation conditions (no, textual, multimodal), the authors measure objective (model simulation, counterfactuals, error detection) and subjective understanding. The findings show explanations do not significantly affect understanding, while aggregation strategy significantly influences understandability, with LMS improving objective understanding and ADD enhancing subjective understanding; counterfactual tasks are particularly challenging. The work suggests that design focus should shift from explanations to the choice of aggregation mechanism and that evaluating understanding benefits from diverse tasks rather than a single metric, with practical implications for developing transparent, user-friendly GRS.

Abstract

Group Recommender Systems (GRS) employing social choice-based aggregation strategies have previously been explored in terms of perceived consensus, fairness, and satisfaction. At the same time, the impact of textual explanations has been examined, but the results suggest a low effectiveness of these explanations. However, user understanding remains fairly unexplored, even if it can contribute positively to transparent GRS. This is particularly interesting to study in more complex or potentially unfair scenarios when user preferences diverge, such as in a minority scenario (where group members have similar preferences, except for a single member in a minority position). In this paper, we analyzed the impact of different types of explanations on user understanding of group recommendations. We present a randomized controlled trial (n = 271) using two between-subject factors: (i) the aggregation strategy (additive, least misery, and approval voting), and (ii) the modality of explanation (no explanation, textual explanation, or multimodal explanation). We measured both subjective (self-perceived by the user) and objective understanding (performance on model simulation, counterfactuals and error detection). In line with recent findings on explanations for machine learning models, our results indicate that more detailed explanations, whether textual or multimodal, did not increase subjective or objective understanding. However, we did find a significant effect of aggregation strategies on both subjective and objective understanding. These results imply that when constructing GRS, practitioners need to consider that the choice of aggregation strategy can influence the understanding of users. Post-hoc analysis also suggests that there is value in analyzing performance on different tasks, rather than through a single aggregated metric of understanding.
Paper Structure (25 sections, 4 figures, 5 tables)

This paper contains 25 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Group Recommendation
  4. Understandability of Recommender Systems
  5. Social Choice-based Explanations
  6. Hypotheses
  7. Methodology
  8. Materials
  9. Group scenarios
  10. Aggregation strategies
  11. Explanations
  12. Variables
  13. Procedure
  14. Sample Size Determination
  15. Results
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Graphical explanation for the Additive Utilitarian (ADD) strategy (example figure). Already visited restaurants are shown in grey. Potential recommendations are colored red. The output of the system is highlighted in yellow. Bar charts for ADD include a horizontal line indicating the sum of all per-item ratings.
  • Figure 2: Survey components in the order as seen by participants. Survey questions in dotted lines were used to derive the dependent variables.
  • Figure 3: Objective (left, 0-1 scale) and subjective (right, 0-7 scale) understanding clustered by aggregation strategy (ADD = additive utilitarian, APP = approval voting, LMS = Least misery) and explanation modality (no_expl = control group, text_expl = textual, Graph_expl = multimodal)
  • Figure 4: Subjective understanding (7-point Likert scale) measured before (preliminary) and after (final) the objective understanding tasks; by explanation modality (no_expl = control group, text_expl = textual, Graph_expl = multimodal)