Fostering human learning is crucial for boosting human-AI synergy

TL;DR

This paper reframes the evidence on human-AI synergy by proposing that human learning is a key, overlooked driver of effective collaboration. Using a re-analysis of 74 studies with robust Bayesian meta-regressions, it shows that outcome feedback tends to increase synergy, and that AI explanations boost this effect when paired with feedback, whereas explanations alone without feedback may reduce synergy. The findings suggest the current literature underestimates the potential of human-AI collaboration and call for research designs that explicitly test and cultivate human learning to enable adaptive, effective partnerships. The authors advocate a paradigm shift toward integrating learning mechanisms into human-AI interaction research and development.

Abstract

The collaboration between humans and artificial intelligence (AI) holds the promise of achieving superior outcomes compared to either acting alone. Nevertheless, our understanding of the conditions that facilitate such human-AI synergy remains limited. A recent meta-analysis showed that, on average, human-AI combinations do not outperform the better individual agent, indicating overall negative human-AI synergy. We argue that this pessimistic conclusion arises from insufficient attention to human learning in the experimental designs used. To substantiate this claim, we re-analyzed all 74 studies included in the original meta-analysis, which yielded two new findings. First, most previous research overlooked design features that foster human learning, such as providing trial-by-trial outcome feedback to participants. Second, our re-analysis, using robust Bayesian meta-regressions, demonstrated that studies providing outcome feedback show relatively higher synergy than those without outcome feedback. Crucially, when feedback is paired with AI explanations we tend to find positive human-AI synergy, while AI explanations provided without feedback were strongly linked to negative synergy, indicating that explanations are useful for synergy only when humans can learn to verify the AI's reliability through feedback. We conclude that the current literature underestimates the potential for human-AI collaboration because it predominantly relies on experimental designs that do not facilitate human learning, thus hindering humans from effectively adapting their collaboration strategies. We therefore advocate for a paradigm shift in human-AI interaction research that explicitly incorporates and tests human learning mechanisms to enhance our understanding of and support for successful human-AI collaboration.

Figures (1)

• Figure 1: (A) Positive human–AI synergy is observed when the joint system performs better than either agent alone. (B) Human--AI interaction without learning opportunities (example: binary classification): (1) Human needs to classify a case as A or B and knows the AI’s classification. (2) Human decides and responds (here with category A). (3) No outcome feedback is provided; thus the human does not know which classification was correct. (C) Human–-AI interaction with learning opportunities: (1) Human needs to classify a case as A or B and knows the AI’s classification and the explanation thereof. Assuming the AI performs well and its explanations are helpful, this focuses the human's attention on the most relevant features. (2) Human decides and responds (here with category A). (3) Human receives outcome feedback (and here learns that the correct response would have been B). (4) Across several cases, this feedback and the AI explanations allow the human to learn about both their own and the AI’s relative skill, both globally and for different kinds of cases. (D) Robust Bayesian model-averaged meta-regression analyses bartos_robust_2025-1 quantifying the evidence that outcome feedback moderates human--AI synergy in the studies reviewed by Vaccaro et al. vaccaro_when_2024. X-axis shows conditional marginal posterior estimates of Hedge's $g$ effect size of human--AI synergy for studies with and without outcome feedback. Dots and whiskers summarize the posterior distribution of the effect-size estimate using the posterior median (point estimate) and 66$\%$ and 95$\%$ quantile credible intervals, respectively (CI; estimation uncertainty). Bayes factors (BF) quantify the relative evidence for the two competing hypotheses that the estimate is non-zero versus zero. Results are presented for all studies, and separately for studies with and without AI explanations.