Human Cognitive Biases in Explanation-Based Interaction: The Case of Within and Between Session Order Effect

TL;DR

This work investigates whether order effects bias human feedback in Explanation-Based Interactive Learning (XIL). Through two large-scale experiments simulating XIL tasks, the authors test within-session and between-session order effects on feedback accuracy, user-model agreement, and trust. They find only limited order effects, with a small within-session primacy influence under difficult conditions and negligible between-session impacts on agreement or perceived trust. The results suggest XIL approaches are robust to order biases in practical debugging scenarios, contributing to understanding human factors in AI and the deployment of explanation-driven model refinement.

Abstract

Explanatory Interactive Learning (XIL) is a powerful interactive learning framework designed to enable users to customize and correct AI models by interacting with their explanations. In a nutshell, XIL algorithms select a number of items on which an AI model made a decision (e.g. images and their tags) and present them to users, together with corresponding explanations (e.g. image regions that drive the model's decision). Then, users supply corrective feedback for the explanations, which the algorithm uses to improve the model. Despite showing promise in debugging tasks, recent studies have raised concerns that explanatory interaction may trigger order effects, a well-known cognitive bias in which the sequence of presented items influences users' trust and, critically, the quality of their feedback. We argue that these studies are not entirely conclusive, as the experimental designs and tasks employed differ substantially from common XIL use cases, complicating interpretation. To clarify the interplay between order effects and explanatory interaction, we ran two larger-scale user studies (n = 713 total) designed to mimic common XIL tasks. Specifically, we assessed order effects both within and between debugging sessions by manipulating the order in which correct and wrong explanations are presented to participants. Order effects had a limited, through significant impact on users' agreement with the model (i.e., a behavioral measure of their trust), and only when examined withing debugging sessions, not between them. The quality of users' feedback was generally satisfactory, with order effects exerting only a small and inconsistent influence in both experiments. Overall, our findings suggest that order effects do not pose a significant issue for the successful employment of XIL approaches. More broadly, our work contributes to the ongoing efforts for understanding human factors in AI.

Figures (8)

• Figure 1: Interface of the user studies (left): participants were asked to evaluate the model's explanation (represented by a bounding box, in blue) of a fictional image classifier. Ideally, this should entirely enclose the face of the person pictured in the image. Participants were instructed to press the "Confirm" button if they deemed the box enclosed the face, or to move it onto the face otherwise, all within a 6 seconds time limit. Right: original images and ground-truth bounding boxes (in red), for reference.
• Figure 2: Schematic illustration of our user studies.
• Figure 3: Average accuracy (top) and agreement (bottom) in Experiment 1 divided by order condition, correctness of model’s placement (correct and incorrect), and image difficulty. Error bars represent standard errors.
• Figure 4: Average accuracy (top) and agreement (bottom) in Experiment 2 divided by order condition, correctness of model’s placement (correct and incorrect), and image difficulty. Error bars represent standard errors.
• Figure 5: Average accuracy in Experiment 1 divided by order condition, correctness of model’s placement (correct, partially wrong, and wrong), and image difficulty. Error bars represent standard errors.