A Scoresheet for Explainable AI
Michael Winikoff, John Thangarajah, Sebastian Rodriguez
TL;DR
The paper addresses the gap between high-level explainability standards and actionable requirements by introducing an XAI scoresheet that captures veracity, global and local explanations, explanatory concepts, question types, and automation. It positions the scoresheet as complementary to IEEE P7001, offering deeper, more actionable guidance for specifying and assessing explainability across diverse AI systems, including multi-agent setups. Through six case studies, the authors demonstrate the scoresheet's versatility, its ability to surface strengths and gaps in explainability, and its potential to inform system selection and regulatory alignment. The work provides a practical framework for stakeholder-oriented explainability design and evaluation, with guidance on operational use and avenues for future refinement and broader validation.
Abstract
Explainability is important for the transparency of autonomous and intelligent systems and for helping to support the development of appropriate levels of trust. There has been considerable work on developing approaches for explaining systems and there are standards that specify requirements for transparency. However, there is a gap: the standards are too high-level and do not adequately specify requirements for explainability. This paper develops a scoresheet that can be used to specify explainability requirements or to assess the explainability aspects provided for particular applications. The scoresheet is developed by considering the requirements of a range of stakeholders and is applicable to Multiagent Systems as well as other AI technologies. We also provide guidance for how to use the scoresheet and illustrate its generality and usefulness by applying it to a range of applications.