Automated Trustworthiness Oracle Generation for Machine Learning Text Classifiers

TL;DR

This work tackles the trustworthiness problem in text classification by introducing TOKI, an automated method to generate trustworthiness oracles from explanations. TOKI constructs class-specific keyword anchors via a four-step keyword identification pipeline and assigns trustworthiness labels by contrasting the semantic relatedness of decision-contributing words to the predicted class. It also proposes a TOKI-guided adversarial attack to reveal vulnerabilities tied to trustworthiness, and establishes a benchmark of ~8,000 predictions across domains to evaluate oracle performance. Empirical results show TOKI outperforms a model-confidence baseline, with robustness across multiple explanations and word-embedding ensembles; the TOKI-guided attack method also yields stronger adversaries with fewer perturbations than the state-of-the-art A2T. Overall, TOKI advances automated trustworthiness testing for ML systems and highlights the potential benefits for SE workflows, model debugging, and safe deployment of text classifiers, while outlining avenues for future exploration of alternative explanations and broader data modalities.

Abstract

Machine learning (ML) for text classification has been widely used in various domains. These applications can significantly impact ethics, economics, and human behavior, raising serious concerns about trusting ML decisions. Studies indicate that conventional metrics are insufficient to build human trust in ML models. These models often learn spurious correlations and predict based on them. In the real world, their performance can deteriorate significantly. To avoid this, a common practice is to test whether predictions are reasonable based on valid patterns in the data. Along with this, a challenge known as the trustworthiness oracle problem has been introduced. Due to the lack of automated trustworthiness oracles, the assessment requires manual validation of the decision process disclosed by explanation methods. However, this is time-consuming, error-prone, and unscalable. We propose TOKI, the first automated trustworthiness oracle generation method for text classifiers. TOKI automatically checks whether the words contributing the most to a prediction are semantically related to the predicted class. Specifically, we leverage ML explanations to extract the decision-contributing words and measure their semantic relatedness with the class based on word embeddings. We also introduce a novel adversarial attack method that targets trustworthiness vulnerabilities identified by TOKI. To evaluate their alignment with human judgement, experiments are conducted. We compare TOKI with a naive baseline based solely on model confidence and TOKI-guided adversarial attack method with A2T, a SOTA adversarial attack method. Results show that relying on prediction uncertainty cannot effectively distinguish between trustworthy and untrustworthy predictions, TOKI achieves 142% higher accuracy than the naive baseline, and TOKI-guided attack method is more effective with fewer perturbations than A2T.

Figures (9)

• Figure 1: LIME explanations of predictions analysing amazon reviews. Words highlighted in orange and blue indicate which class, negative or positive, they contribute to, with the shade representing their importance.
• Figure 2: LIME explanations of predictions analysing amazon reviews. Bold words represent perturbations, while words highlighted in orange and blue indicate their contribution to the negative or positive class, with the shade reflecting their importance.
• Figure 3: Comparison between two ML system development processes: without and with trustworthiness testing. Blue lines indicate the differences between these processes.
• Figure 4: The process of TOKI, with blue and yellow background colors indicating the first and second pipelines.
• Figure 5: Adversarial examples generated by TOKI (Ours) and A2T. Words highlighted in orange or blue show their contribution to the negative or positive class, with the shade indicating their level of importance.

Theorems & Definitions (5)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Definition 5