U-Trustworthy Models.Reliability, Competence, and Confidence in Decision-Making

TL;DR

The paper introduces a competence-based framework for trust in predictive models, defining $\mathcal{U}$-trustworthiness as the ability to maximize Bayes utility within a specified task subset. It formalizes the triad of Reliance, Competency, and Confidence, and proves that properly-ranked classifiers attain $\mathcal{U}$-trustworthiness across cost-sensitive and equity-aware utilities, with Bayes optimality tying max utility to trustworthiness. AUC is advocated as the primary measure of trustworthiness since it aligns with ranking-based utility maximization, and extensive empirical results across multiple datasets show AUC-guided model selection and hyper-parameter tuning yield higher expected utility than alternative metrics. The work also critically examines calibration as a sole prerequisite for trust and delineates distinctions from fairness, outlining limitations and future avenues for extending the framework to broader tasks and multi-class settings.

Abstract

With growing concerns regarding bias and discrimination in predictive models, the AI community has increasingly focused on assessing AI system trustworthiness. Conventionally, trustworthy AI literature relies on the probabilistic framework and calibration as prerequisites for trustworthiness. In this work, we depart from this viewpoint by proposing a novel trust framework inspired by the philosophy literature on trust. We present a precise mathematical definition of trustworthiness, termed $\mathcal{U}$-trustworthiness, specifically tailored for a subset of tasks aimed at maximizing a utility function. We argue that a model's $\mathcal{U}$-trustworthiness is contingent upon its ability to maximize Bayes utility within this task subset. Our first set of results challenges the probabilistic framework by demonstrating its potential to favor less trustworthy models and introduce the risk of misleading trustworthiness assessments. Within the context of $\mathcal{U}$-trustworthiness, we prove that properly-ranked models are inherently $\mathcal{U}$-trustworthy. Furthermore, we advocate for the adoption of the AUC metric as the preferred measure of trustworthiness. By offering both theoretical guarantees and experimental validation, AUC enables robust evaluation of trustworthiness, thereby enhancing model selection and hyperparameter tuning to yield more trustworthy outcomes.

Figures (3)

• Figure 1: Performance of three models under different criteria. Top row: Calibration plot (left). Utility comparison using 0-1 loss (middle) as a function of decision threshold. Utility comparison using a general utility function as a function of decision threshold (right). The confidence bands represent the 16th and 84th percentiles of 400 data realizations. Bottom row: Distribution of Brier score (left), Accuracy(middle), NetTrustScore (right) under 400 data realizations.
• Figure 2: Left: Average calibration curve. Right: The performance of Logistic Regression, selected based on accuracy/Brier score, and Random Forest, chosen based on AUC, for a class of utility functions specified by parameter $c$, cost coefficient. The average maximum utility on the test sample for 20 random test/train realizations and the shaded region is 68% error on the mean.
• Figure 3: Hyper-parameter Tuning of k-NN. Left: Average cross-validation performance vs. n_neighbors. The optimal $k$ for (accuracy) AUC is (150) 200, indicated by the dotted horizontal line. Right: Utility as a function of decision threshold for $k = 150$ (red curve) and $k = 200$ (blue curve). The shaded region represents the standard error on mean based on 200 random test/train realizations.

Theorems & Definitions (41)

• Definition 1: $\mathcal{U}$-Trustworthy
• Definition 2
• Proposition 1
• Theorem 1
• Definition 3
• Definition 4: Properly-Ranked Classifier
• Theorem 2: $\mathcal{U}$-Competency Theorem
• Proposition 2
• Definition 5
• Example 1