Mitigating subjectivity and bias in AI development indices: A robust approach to redefining country rankings

TL;DR

This work addresses subjectivity and bias in AI development indices by integrating a robust MCDA framework that combines the Choquet integral with SMAA to account for intercriteria interactions and weight uncertainty. It applies two unsupervised methods, $u1$ and $u2$, to learn Shapley interaction indices and compares Choquet-based rankings $CI^{u1}$ and $CI^{u2}$ against the GAII baseline, using Kendall distance to quantify differences. A robust ranking is derived by pairing SMAA with the Condorcet method, yielding three final rankings: $WS$-Cond., $CI^{u2}$-Cond., and $CI^{u1}$-Cond., which show reduced sensitivity to weight changes and bias mitigation, with $CI^{u2}$ often closer to GAII. The results demonstrate significant correlations among AI dimensions, particularly $g_5$ (Research), and indicate that the proposed approach improves robustness and reduces subjectivity, offering practical implications for policy decisions and international comparisons of AI readiness.

Abstract

Countries worldwide have been implementing different actions national strategies for Artificial Intelligence (AI) to shape policy priorities and guide their development concerning AI. Several AI indices have emerged to assess countries' progress in AI development, aiding decision-making on investments and policy choices. Typically, these indices combine multiple indicators using linear additive methods such as weighted sums, although they are limited in their ability to account for interactions among indicators. Another limitation concerns the use of deterministic weights, which can be perceived as subjective and vulnerable to debate and scrutiny, especially by nations that feel disadvantaged. Aiming at mitigating these problems, we conduct a methodological analysis to derive AI indices based on multiple criteria decision analysis. Initially, we assess correlations between different AI dimensions and employ the Choquet integral to model them. Thus, we apply the Stochastic Multicriteria Acceptability Analysis (SMAA) to conduct a sensitivity analysis using both weighted sum and Choquet integral in order to evaluate the stability of the indices with regard the weights. Finally, we introduce a novel ranking methodology based on SMAA, which considers several sets of weights to derive the ranking of countries. As a result, instead of using predefined weights, in the proposed approach, the ranking is achieved based on the probabilities of countries in occupying a specific position. In the computational analysis, we utilize the data employed in The Global AI Index proposed by Tortoise. Results reveal correlations in the data, and our approach effectively mitigates bias. In the sensitivity analysis, we scrutinize changes in the ranking resulting from weight adjustments. We demonstrate that our proposal rankings closely align with those derived from weight variations, proving to be more robust.

Figures (5)

• Figure 1: Kendall tau distances to measure differences between rankings.
• Figure 2: SMAA acceptability indices considering total absence of weight information.
• Figure 3: Visualization of variability in Kendall tau's distance of the different rankings considering total absence of weight information.
• Figure 4: SMAA Acceptability indices considering preference order of weights.
• Figure 5: Kendall tau's distance for ordered weights.