Explanation User Interfaces: A Systematic Literature Review

TL;DR

This systematic literature review analyzes Explanation User Interfaces (XUIs) to understand how explanations from XAI are effectively presented to users. It integrates algorithmic explanation techniques, interactive design, evaluation methods, and design guidelines into a cohesive framework, culminating in the HERMES platform to guide practitioners. The study reveals prevalent use of neural networks with feature-importance, counterfactuals, and SHAP across visual, textual, and interactive modalities, and highlights the importance of user-centered design, contextual information, and multi-level visualizations. The findings offer practical guidance for designing trustworthy, usable XUIs in high-stakes domains and identify future challenges, including co-design adoption and managing emergent AI capabilities like LLMs.

Abstract

Artificial Intelligence (AI) is one of the major technological advancements of this century, bearing incredible potential for users through AI-powered applications and tools in numerous domains. Being often black-box (i.e., its decision-making process is unintelligible), developers typically resort to eXplainable Artificial Intelligence (XAI) techniques to interpret the behaviour of AI models to produce systems that are transparent, fair, reliable, and trustworthy. However, presenting explanations to the user is not trivial and is often left as a secondary aspect of the system's design process, leading to AI systems that are not useful to end-users. This paper presents a Systematic Literature Review on Explanation User Interfaces (XUIs) to gain a deeper understanding of the solutions and design guidelines employed in the academic literature to effectively present explanations to users. To improve the contribution and real-world impact of this survey, we also present a framework for Human-cEnteRed developMent of Explainable user interfaceS (HERMES) to guide practitioners and academics in the design and evaluation of XUIs.

Figures (8)

• Figure 1: PRISMA Page2021PRISMA flow diagram depicting the identification, screening, eligibility, and inclusion process of studies in the systematic literature review.
• Figure 2: Heatmap showing the relationship between user types and application domains, illustrating how different user categories are distributed across various domains
• Figure 3: Frequency of visualisation techniques used in explanation user interfaces. The bar chart presents the types of visualisations employed to convey explanations, with heatmaps, bar charts, and trend lines being the most frequently used. A wide range of other formats---such as scatter plots, histograms, Sankey diagrams, and tree structures---are also represented, reflecting the diversity of visual explanation strategies.
• Figure 4: Methods used to evaluate explanation interfaces across different user types. The heatmap shows the distribution of evaluation methods such as controlled experiments, interviews, surveys, and usability studies, categorised by the type of user involved (e.g., AI experts, domain experts, non-experts). Controlled experiments and interviews are the most common methods, particularly among non-experts and domain experts, reflecting a strong emphasis on both empirical validation and qualitative feedback in XUI evaluation.
• Figure 5: Reported evaluation criteria by user type. The stacked bar chart illustrates which user groups (AI experts, domain experts, non-experts, and unspecified) were associated with various evaluation criteria used in assessing explanation interfaces. Criteria such as trust, understandability, usability, and perceived effectiveness were evaluated across all user types, highlighting the multidimensional nature of user-centered evaluation in explainable systems. Notably, transparency was not evaluated in studies involving AI experts.