Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary

TL;DR

This paper quantitatively model how humans integrate both AI recommendations and explanations into their decision process, and whether this quantitative understanding of human behavior from the learned model can be utilized to manipulate AI explanations, thereby nudging individuals towards making targeted decisions.

Abstract

Recent advances in AI models have increased the integration of AI-based decision aids into the human decision making process. To fully unlock the potential of AI-assisted decision making, researchers have computationally modeled how humans incorporate AI recommendations into their final decisions, and utilized these models to improve human-AI team performance. Meanwhile, due to the ``black-box'' nature of AI models, providing AI explanations to human decision makers to help them rely on AI recommendations more appropriately has become a common practice. In this paper, we explore whether we can quantitatively model how humans integrate both AI recommendations and explanations into their decision process, and whether this quantitative understanding of human behavior from the learned model can be utilized to manipulate AI explanations, thereby nudging individuals towards making targeted decisions. Our extensive human experiments across various tasks demonstrate that human behavior can be easily influenced by these manipulated explanations towards targeted outcomes, regardless of the intent being adversarial or benign. Furthermore, individuals often fail to detect any anomalies in these explanations, despite their decisions being affected by them.

Figures (4)

• Figure 1: Comparing averageFPRD and FNRD of the human decision outcomes under the adversarially manipulated explanation, SHAP explanation, or LIME explanation. Error bars represent the 95% confidence intervals of the mean values. $\textsuperscript{*}$, $\textsuperscript{**}$, and $\textsuperscript{***}$ denote significance levels of $0.1$, $0.05$, and $0.01$, respectively. For both FPRD and FNRD, a value closer to zero indicates that the human decisions are more fair.
• Figure 2: Comparing the average human perceived transparency and usefulness of the adversarially manipulated explanation, SHAP explanation, and LIME explanation. Error bars represent the 95% confidence intervals of the mean values.
• Figure 3: Comparing the average accuracy, overreliance, and the underreliance of human decision outcomes under the benignly manipulated explanation, SHAP explanation, or LIME explanation. Error bars represent the 95% confidence intervals of the mean values. $\textsuperscript{*}$, $\textsuperscript{**}$, and $\textsuperscript{***}$ denote significance levels of $0.1$, $0.05$, and $0.01$ respectively.
• Figure 4: Comparing the average human perceived transparency and usefulness of the benignly manipulated explanations, SHAP explanations, and LIME explanations. Error bars represent the 95% confidence intervals of the mean values.