Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making

TL;DR

The paper investigates how second opinions influence human-AI decision-making, addressing whether advisors from peers or AI can curb misaligned reliance on AI. Through three pre-registered MTurk experiments using a RoBERTa-based sentiment classifier and 20 sentiment tasks, it shows that always-presented second opinions reduce AI over-reliance but increase under-reliance, with effects modulated by agreement level rather than source. Allowing decision-makers to solicit second opinions on demand partially mitigates over-reliance without consistently elevating under-reliance, particularly when opinions strongly agree with the AI. The findings highlight nuanced design considerations for integrating second opinions into AI-assisted workflows and emphasize the trade-offs between reliability, cognitive load, and decision speed. Overall, second opinions hold promise for improving human-AI collaboration, but their impact hinges on presentation format, agreement dynamics, and the potential costs of obtaining second opinions.

Abstract

AI assistance in decision-making has become popular, yet people's inappropriate reliance on AI often leads to unsatisfactory human-AI collaboration performance. In this paper, through three pre-registered, randomized human subject experiments, we explore whether and how the provision of {second opinions} may affect decision-makers' behavior and performance in AI-assisted decision-making. We find that if both the AI model's decision recommendation and a second opinion are always presented together, decision-makers reduce their over-reliance on AI while increase their under-reliance on AI, regardless whether the second opinion is generated by a peer or another AI model. However, if decision-makers have the control to decide when to solicit a peer's second opinion, we find that their active solicitations of second opinions have the potential to mitigate over-reliance on AI without inducing increased under-reliance in some cases. We conclude by discussing the implications of our findings for promoting effective human-AI collaborations in decision-making.

Figures (8)

• Figure 1: An illustration of our task interface in Experiment 1. In this example, the treatment is presenting both the AI model's prediction (positive) and the second opinion from a peer worker (negative).
• Figure 2: The effects of second opinions from human peers on subjects' overall reliance (\ref{['fig:exp1:reliance']}), over-reliance (\ref{['fig:exp1:overreliance']}), under-reliance (\ref{['fig:exp1:underreliance']}), and appropriate reliance (\ref{['fig:exp1:approp_reliance']}) on the AI model across treatments. Error bars represent the standard errors of the mean.
• Figure 3: The effects of second opinions from human peers on subjects' decision time (\ref{['fig:exp1:time']}), confidence in their correct decisions (\ref{['fig:exp1:confidence_correct']}), and confidence in their incorrect decisions (\ref{['fig:exp1:confidence_incorrect']}) across treatments. Error bars represent the standard errors of the mean.
• Figure 4: The effects of second opinions from different sources (i.e., human peers or another AI model) and with different levels of agreement with the primary AI model on subjects' overall reliance (\ref{['fig:exp3:reliance']}), over-reliance (\ref{['fig:exp3:overreliance']}), under-reliance (\ref{['fig:exp3:underreliance']}), and appropriate reliance (\ref{['fig:exp3:approp_reliance']}) on AI. Error bars and error shades represent the standard errors of the mean.
• Figure 5: An example of our task interface in Experiment 3 (for treatments where subjects can solicit the second opinions), before (\ref{['fig:exp2:interface_before']}) and after (\ref{['fig:exp2:interface_after']}) subjects clicked the "Request" button.