A Sim2Real Approach for Identifying Task-Relevant Properties in Interpretable Machine Learning

TL;DR

XAIsim2real presents a sim2real workflow that systematically links explanation properties (faithfulness, robustness, sparsity) to downstream decision tasks. By generating hypotheses in synthetic proxy-user studies (across forbidden features, counterfactual simulation, and forward simulation) and then validating them with real users, the approach demonstrates which properties matter for specific tasks and how cognitive budgets modulate usefulness. The results show faithful explanations excel for some tasks while robustness or sparsity become advantageous in others, and that proxy-based predictions reasonably anticipate real-user behavior. This framework offers a scalable, generalizable method to optimize explanations before costly human studies and can be extended to a broader range of human-AI decision-making contexts.

Abstract

Explanations of an AI's function can assist human decision-makers, but the most useful explanation depends on the decision's context, referred to as the downstream task. User studies are necessary to determine the best explanations for each task. Unfortunately, testing every explanation and task combination is impractical, especially considering the many factors influencing human+AI collaboration beyond the explanation's content. This work leverages two insights to streamline finding the most effective explanation. First, explanations can be characterized by properties, such as faithfulness or complexity, which indicate if they contain the right information for the task. Second, we introduce XAIsim2real, a pipeline for running synthetic user studies. In our validation study, XAIsim2real accurately predicts user preferences across three tasks, making it a valuable tool for refining explanation choices before full studies. Additionally, it uncovers nuanced relationships, like how cognitive budget limits a user's engagement with complex explanations -- a trend confirmed with real users.

Figures (12)

• Figure 1: Graphical overview of XAIsim2real, which simulates a user study in three main steps. First, XAIsim2real forms the property-optimized explanations. Second, the user proxy is trained to perform a task on a set of example inputs $\mathbf{x}_h$ and correct decisions $y^*_h$ (the property-optimized explanations are part of the human-inputs). Third, the user proxy is evaluated on a set of test inputs, from which we compute the overall task-performance.
• Figure 2: Demonstrative, one-dimensional example of $f_\text{trend+wiggle}$, and how explanations with different properties would describe it differently. This function (in black) has a clear long-term linear trend with short-term oscillations on top. Explanations are represented as lines at input points of interest ('X'). Faithful explanations (red) will capture the short-term oscillations. Sparse explanations (yellow) will also do this, but will also return '0' at points, to maintain sparsity. Robust explanations (blue) will capture the long-term trend but not the oscillations. Sparse+robust explanations (green) are zeroed-out for this dimension, but can be non-zero for other dimensions.
• Figure 3: Simulation results. Each subplot is a task and each grouping represents results within a setting. We validate the settings highlighted in yellow with user studies. Error bars are 95% confidence intervals over the different proxies that result from resampling the $10$ training points.
• Figure 4: The proxy users used the same $10$ training points and $30$ test points as the real users. Error bars are 95% confidence intervals across training the model (randomness due to the initialization of the decision tree optimization).
• Figure 5: Example UI for forward simulation task, where participants diagnosed alien as healthy or sick. The explanation (covered as helpful information from an alien researcher) is from a sparse explanation. UIs for the remaining tasks are in \ref{['appendix: screenshots']}.