Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Explanations Through LLMs: Beyond Traditional User Studies

Francesco Bombassei De Bona, Gabriele Dominici, Tim Miller, Marc Langheinrich, Martin Gjoreski

TL;DR

The use of Large Language Models (LLMs) to replicate human participants to help streamline XAI evaluation is explored, suggesting that LLMs could provide a scalable and cost-effective way to simplify qualitative XAI evaluation.

Abstract

As AI becomes fundamental in sectors like healthcare, explainable AI (XAI) tools are essential for trust and transparency. However, traditional user studies used to evaluate these tools are often costly, time consuming, and difficult to scale. In this paper, we explore the use of Large Language Models (LLMs) to replicate human participants to help streamline XAI evaluation. We reproduce a user study comparing counterfactual and causal explanations, replicating human participants with seven LLMs under various settings. Our results show that (i) LLMs can replicate most conclusions from the original study, (ii) different LLMs yield varying levels of alignment in the results, and (iii) experimental factors such as LLM memory and output variability affect alignment with human responses. These initial findings suggest that LLMs could provide a scalable and cost-effective way to simplify qualitative XAI evaluation.

Evaluating Explanations Through LLMs: Beyond Traditional User Studies

TL;DR

The use of Large Language Models (LLMs) to replicate human participants to help streamline XAI evaluation is explored, suggesting that LLMs could provide a scalable and cost-effective way to simplify qualitative XAI evaluation.

Abstract

As AI becomes fundamental in sectors like healthcare, explainable AI (XAI) tools are essential for trust and transparency. However, traditional user studies used to evaluate these tools are often costly, time consuming, and difficult to scale. In this paper, we explore the use of Large Language Models (LLMs) to replicate human participants to help streamline XAI evaluation. We reproduce a user study comparing counterfactual and causal explanations, replicating human participants with seven LLMs under various settings. Our results show that (i) LLMs can replicate most conclusions from the original study, (ii) different LLMs yield varying levels of alignment in the results, and (iii) experimental factors such as LLM memory and output variability affect alignment with human responses. These initial findings suggest that LLMs could provide a scalable and cost-effective way to simplify qualitative XAI evaluation.

Paper Structure

This paper contains 19 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Traditional user studies to evaluate XAI tools
  3. Can LLMs evaluate XAI tools?
  4. Experiments
  5. Evaluation Setting
  6. User study
  7. Estimating human preferences with LLMs
  8. Metrics and statistical tests
  9. Results
  10. Evaluating XAI tools with LLMs partially mirrors user study conclusions. (Figure \ref{['fig:overall-heatmap']})
  11. LLMs exhibit slight differences in absolute alignment with human preferences. (Figure \ref{['fig:overall-mse-qwen']})
  12. Different LLMs exhibit varying levels of general alignment with human responses. (Figure \ref{['fig:overall-heatmap']})
  13. Usage of memory impacts LLMs’ general alignment. (Figure \ref{['fig:factor-comparison']})
  14. Simulating different users through aggregation leads to opposite results. (Figure \ref{['fig:factor-comparison']})
  15. Limitations
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Our vision in comparing Human and LLM Evaluators for XAI Tool Effectiveness
  • Figure 2: Concordance of the results for each statistical test. Concordance is computed by merging the results of the ANOVA test and the results of the comparison of the averaged values.
  • Figure 3: Bar charts representing the MSE between users and Qwen2-72B in the three tasks
  • Figure 4: Concordance of the results for each statistical test aggregated by experimental settings. Experimental settings explored comprise conversation with or without conversation memory and usage of aggregated inference runs.