Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Navigating AI Fallibility: Examining People's Reactions and Perceptions of AI after Encountering Personality Misrepresentations

Qiaosi Wang, Chidimma L. Anyi, Vedant Das Swain, Ashok K. Goel

TL;DR

This work investigates how people react to and perceive AI after encountering personality misrepresentations in hyper-personalized team-matching AI within a higher-education context. Using a mixed-methods design with a Wizard-of-Oz SAMI and two conditions (accurate vs inaccurate inferences), the authors reveal that users' existing and evolving AI knowledge (AI literacy) significantly shapes changes in trust, while social perceptions are less malleable. They identify three rationales—SAMI as a machine, a human, or magic—that people adopt to explain AI behavior, and link these to reactions of over-trusting, rationalizing, and forgiving. The findings inform design directions for mitigation and repair strategies that account for users’ AI knowledge and evolving mental models to reduce harms from AI fallibility. Overall, the paper contributes to understanding how AI literacy moderates trust dynamics after AI mistakes and proposes knowledge-aware approaches for responsible AI design in educational settings.

Abstract

Many hyper-personalized AI systems profile people's characteristics (e.g., personality traits) to provide personalized recommendations. These systems are increasingly used to facilitate interactions among people, such as providing teammate recommendations. Despite improved accuracy, such systems are not immune to errors when making inferences about people's most personal traits. These errors manifested as AI misrepresentations. However, the repercussions of such AI misrepresentations are unclear, especially on people's reactions and perceptions of the AI. We present two studies to examine how people react and perceive the AI after encountering personality misrepresentations in AI-facilitated team matching in a higher education context. Through semi-structured interviews (n=20) and a survey experiment (n=198), we pinpoint how people's existing and newly acquired AI knowledge could shape their perceptions and reactions of the AI after encountering AI misrepresentations. Specifically, we identified three rationales that people adopted through knowledge acquired from AI (mis)representations: AI works like a machine, human, and/or magic. These rationales are highly connected to people's reactions of over-trusting, rationalizing, and forgiving of AI misrepresentations. Finally, we found that people's existing AI knowledge, i.e., AI literacy, could moderate people's changes in their trust in AI after encountering AI misrepresentations, but not changes in people's social perceptions of AI. We discuss the role of people's AI knowledge when facing AI fallibility and implications for designing responsible mitigation and repair strategies.

Navigating AI Fallibility: Examining People's Reactions and Perceptions of AI after Encountering Personality Misrepresentations

TL;DR

This work investigates how people react to and perceive AI after encountering personality misrepresentations in hyper-personalized team-matching AI within a higher-education context. Using a mixed-methods design with a Wizard-of-Oz SAMI and two conditions (accurate vs inaccurate inferences), the authors reveal that users' existing and evolving AI knowledge (AI literacy) significantly shapes changes in trust, while social perceptions are less malleable. They identify three rationales—SAMI as a machine, a human, or magic—that people adopt to explain AI behavior, and link these to reactions of over-trusting, rationalizing, and forgiving. The findings inform design directions for mitigation and repair strategies that account for users’ AI knowledge and evolving mental models to reduce harms from AI fallibility. Overall, the paper contributes to understanding how AI literacy moderates trust dynamics after AI mistakes and proposes knowledge-aware approaches for responsible AI design in educational settings.

Abstract

Many hyper-personalized AI systems profile people's characteristics (e.g., personality traits) to provide personalized recommendations. These systems are increasingly used to facilitate interactions among people, such as providing teammate recommendations. Despite improved accuracy, such systems are not immune to errors when making inferences about people's most personal traits. These errors manifested as AI misrepresentations. However, the repercussions of such AI misrepresentations are unclear, especially on people's reactions and perceptions of the AI. We present two studies to examine how people react and perceive the AI after encountering personality misrepresentations in AI-facilitated team matching in a higher education context. Through semi-structured interviews (n=20) and a survey experiment (n=198), we pinpoint how people's existing and newly acquired AI knowledge could shape their perceptions and reactions of the AI after encountering AI misrepresentations. Specifically, we identified three rationales that people adopted through knowledge acquired from AI (mis)representations: AI works like a machine, human, and/or magic. These rationales are highly connected to people's reactions of over-trusting, rationalizing, and forgiving of AI misrepresentations. Finally, we found that people's existing AI knowledge, i.e., AI literacy, could moderate people's changes in their trust in AI after encountering AI misrepresentations, but not changes in people's social perceptions of AI. We discuss the role of people's AI knowledge when facing AI fallibility and implications for designing responsible mitigation and repair strategies.
Paper Structure (41 sections, 3 equations, 8 figures, 4 tables)

This paper contains 41 sections, 3 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. People's Perceptions and Reactions of Hyper-Personalized AI
  4. Responding and Mitigating AI Fallibilities
  5. Study Overview
  6. Study 1: Understanding Students' Perceptions and Reactions to AI Misrepresentation
  7. Study 1 Method
  8. Study 1 Participant and Recruitment
  9. Fabricating SAMI Inferences
  10. Study 1 Remote User Study Session
  11. Study 1 Data Analysis
  12. Study 1 Findings: Interpreting and Reacting to AI after Encountering AI (Mis)representation
  13. Interpreting SAMI: Machine? Human? Or Magic?
  14. Reacting to AI Misrepresentation: Over-trusting, Rationalizing, and Forgiving
  15. Study 2: Examining the Changes in Students' Perception of AI after Encountering AI Misrepresentations
  16. ...and 26 more sections

Figures (8)

  • Figure 1: Study flow diagram that shows the procedures of Study 1 and Study 2. Study 2 occurred after Study 1 was concluded. All personal inferences shown to participants were either accurate or inaccurate based on the condition assigned to the participants.
  • Figure 2: (a) AI literacy significantly moderated the effect of AI misrepresentations on students' changes in overall trust of SAMI. (b) (c) (d) show that AI literacy does not significantly moderate the effect of AI misrepresentations on students' changes in perceived intelligence, anthropomorphism, and likeability of SAMI.
  • Figure A1: This figure shows the sample and our inference fabrication process for the sample student. The top half of this figure shows one of the samples we showed to the participants that is inaccurate. The bottom half of this figure shows how we utilized participants' personality ground truth filled out in the preliminary survey to fabricate inferences for them based on the condition they were assigned.
  • Figure D2: This is a screenshot of the website that we built for participants in Study 2 to retrieve SAMI's inferences about them by entering their Prolific ID.
  • Figure F3: Density plots visualizing the participant distribution of changes in overall trust, intelligence, anthropomorphism, and likeability in the accurate and inaccurate conditions.
  • ...and 3 more figures