Just Like Me: The Role of Opinions and Personal Experiences in The Perception of Explanations in Subjective Decision-Making

TL;DR

The paper addresses how subjective decision-making can be shaped by explanations that contain personal opinions and experiences, exploring whether LLM-generated explanations can surface diverse viewpoints. Using a twenty-participant interview study with subtle sexism scenarios, it compares human and AI explanations and analyzes perceived authorship, relevance, convincingness, and trustworthiness. Key findings indicate that opinions and experiences within explanations boost perceived persuasiveness and trust, particularly when aligned with the user's own beliefs, but also amplify confirmation bias and attribution concerns. The work highlights ethical challenges and design considerations for collaborative AI systems, suggesting multiple-perspective outputs and transparent authorship to balance novelty with trust. This has practical implications for deploying AI in subjective decision contexts and calls for trust-calibration strategies and careful presentation of AI-generated experiential content.

Abstract

As large language models (LLMs) advance to produce human-like arguments in some contexts, the number of settings applicable for human-AI collaboration broadens. Specifically, we focus on subjective decision-making, where a decision is contextual, open to interpretation, and based on one's beliefs and values. In such cases, having multiple arguments and perspectives might be particularly useful for the decision-maker. Using subtle sexism online as an understudied application of subjective decision-making, we suggest that LLM output could effectively provide diverse argumentation to enrich subjective human decision-making. To evaluate the applicability of this case, we conducted an interview study (N=20) where participants evaluated the perceived authorship, relevance, convincingness, and trustworthiness of human and AI-generated explanation-text, generated in response to instances of subtle sexism from the internet. In this workshop paper, we focus on one troubling trend in our results related to opinions and experiences displayed in LLM argumentation. We found that participants rated explanations that contained these characteristics as more convincing and trustworthy, particularly so when those opinions and experiences aligned with their own opinions and experiences. We describe our findings, discuss the troubling role that confirmation bias plays, and bring attention to the ethical challenges surrounding the AI generation of human-like experiences.

Figures (1)

• Figure 1: Overview of the research methodology