Rules or Weights? Comparing User Understanding of Explainable AI Techniques with the Cognitive XAI-Adaptive Model

TL;DR

The work addresses how users interpret explanatory XAI schemas and proposes CoXAM, a cognitive model that fuses memory chunks, drift-diffusion decision dynamics, and computational rationality to select among rule-based and weight-based explanations across forward and counterfactual tasks. Through formative and summative studies, it identifies distinct reasoning strategies and demonstrates that CoXAM better predicts human decisions than baseline proxies while reproducing key empirical findings (e.g., counterfactual tasks are harder; rules are harder to recall). The model is validated on two tabular datasets (Wine Quality and Mushroom Edibility) and shows dataset-dependent preferences for explanation types, enabling targeted debugging and benchmarking of XAI methods. Overall, CoXAM offers a principled, interpretable framework to simulate and optimize human understanding in XAI applications, with implications for UI design, tutorials, and future multimodal explanations.

Abstract

Rules and Weights are popular XAI techniques for explaining AI decisions. Yet, it remains unclear how to choose between them, lacking a cognitive framework to compare their interpretability. In an elicitation user study on forward and counterfactual decision tasks, we identified 7 reasoning strategies of interpreting three XAI Schemas - weights, rules, and their hybrid. To analyze their capabilities, we propose CoXAM, a Cognitive XAI-Adaptive Model with shared memory representation to encode instance attributes, linear weights, and decision rules. CoXAM employs computational rationality to choose among reasoning processes based on the trade-off in utility and reasoning time, separately for forward or counterfactual decision tasks. In a validation study, CoXAM demonstrated a stronger alignment with human decision-making compared to baseline machine learning proxy models. The model successfully replicated and explained several key empirical findings, including that counterfactual tasks are inherently harder than forward tasks, decision tree rules are harder to recall and apply than linear weights, and the helpfulness of XAI depends on the application data context, alongside identifying which underlying reasoning strategies were most effective. With CoXAM, we contribute a cognitive basis to accelerate debugging and benchmarking disparate XAI techniques.

Figures (32)

• Figure 1: Overall approach to model users in XAI understanding across three studies: I) Formative study to elicit reasoning strategies, II) Modeling study to examine user reasoning, and III) Summative study to observe decisions.
• Figure 2: Weight-based XAI with Tabular UI components for: a) Attribute names of the instance. b) Value of each attribute with bar to indicate how high/low. c) Factors to multiply with each attribute value. d) Partial contributions of each attribute based on value $x$ factor. e) Prediction of Weights explanation for current instance. f) Prediction of AI. a--b are always shown; c is only shown in trials with Weights explanation; d, e, f is not shown for decision testing.
• Figure 3: Rule-based XAI with Decision Tree UI for a) low (wine quality) and b) high complexity (mushrooms). The black outline highlights the path that corresponds to the current instance values.
• Figure 4: User performance of forward simulation task in a), and b) and counterfactual simulation task in c), and d), across datasets: Wine Quality (left), and Mushrooms (right) across XAI Schemas, and testing conditions (w/o XAI, w/ XAI). Error bars are 95% CIs. Random performance for the forward simulation task would be 50%.
• Figure 5: Results of summative user study (orange) compared to CoXAM (blue) of accuracy in the forward simulation task. Error bars are 95% CI.