The hybrid confirmation tree: A robust strategy for hybrid intelligence

TL;DR

The hybrid confirmation tree emerges as a practical, efficient, and robust strategy for hybrid collective intelligence that maintains human agency that provides greater flexibility in navigating true and false positive trade-offs compared to fixed human-only heuristics like hierarchies and polyarchies.

Abstract

Combining human and artificial intelligence (AI) is a potentially powerful approach to boost decision accuracy. However, few such approaches exist that effectively integrate both types of intelligence while maintaining human agency. Here, we introduce and evaluate the hybrid confirmation tree, a simple aggregation strategy that compares the independent decisions of both a human and AI, with disagreements triggering a second human tiebreaker. Through analytical derivations, we show that the hybrid confirmation tree can match and exceed the accuracy of a three-person human majority vote while requiring fewer human inputs, particularly when AI accuracy is comparable to or exceeds human accuracy. We analytically demonstrate that the hybrid confirmation tree's ability to achieve complementarity -- outperforming individual humans, AI, and the majority vote -- is maximized when human and AI accuracies are similar and their decisions are not overly correlated. Empirical reanalysis of six real-world datasets (covering skin cancer diagnosis, deepfake detection, geopolitical forecasting, and criminal rearrest) validates these findings, showing that the hybrid confirmation tree improves accuracy over the majority vote by up to 10 percentage points while reducing the cost of decision making by 28--44$\%$. Furthermore, the hybrid confirmation tree provides greater flexibility in navigating true and false positive trade-offs compared to fixed human-only heuristics like hierarchies and polyarchies. The hybrid confirmation tree emerges as a practical, efficient, and robust strategy for hybrid collective intelligence that maintains human agency.

Figures (13)

• Figure 1: The hybrid confirmation tree procedure. The independent decisions of a human and AI are compared. In cases of agreement, that decision is accepted. In cases of disagreement, a second human breaks the tie. The final decision is therefore always approved by at least one human.
• Figure 1: Comparison of the accuracy of the hybrid confirmation tree, single humans, a machine, and a three-person majority vote as a function of human accuracy and machine accuracy. Regions where the hybrid confirmation tree outperforms only the majority vote (blue), only the machine (yellow), and both the majority vote and the machine (dark blue), as well as a region where the hybrid confirmation tree performs worse than the majority vote and the machine alone (gray).
• Figure 2: Comparing the accuracy of the hybrid confirmation tree, a machine alone or humans alone, and a three-person majority vote as a function of human accuracy and machine accuracy. (A) Regions where the hybrid confirmation tree outperforms only the majority vote (blue), only the machine (yellow), and both the majority vote and the machine (dark blue). (B) Results shown in (A) but for varying levels of human--machine (HM; $\kappa_{HM}$) and human--human (HH; $\kappa_{HH}$) dependence. Note that high $\kappa$-values are only possible for more similar human and machine accuracies.
• Figure 2: Comparison of the hybrid confirmation tree, single humans, a machine, and a three-person majority vote as a function of human accuracy and machine accuracy. Colors indicate regions where the hybrid confirmation tree outperforms the human majority vote (blue), the machine (yellow), and both the human majority vote and the machine (dark blue), and where it performs worse than the machine and the human majority vote (gray). Columns and rows show increasing human--human ($\kappa_{HH}$) and human--machine ($\kappa_{HM}$) decision making correlation. Higher $\kappa$ values are only possible for more similar human and machine accuracies (i.e., large differences between human and machine accuracies result in lower $\kappa$ values).
• Figure 3: The accuracy of single humans, a three-person majority vote, the hybrid confirmation tree (HCT), and a machine, per domain. Results were generated by selecting the threshold setting that maximized hybrid confirmation tree accuracy. Dashed horizontal lines indicate chance level (i.e., accuracy of 0.5). Brackets indicate the relative reduction in human decision makers of the hybrid confirmation tree compared to the human majority vote.