Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards a Responsible AI Metrics Catalogue: A Collection of Metrics for AI Accountability

Boming Xia, Qinghua Lu, Liming Zhu, Sung Une Lee, Yue Liu, Zhenchang Xing

TL;DR

This work tackles the gap between high-level Responsible AI principles and actionable accountability guidance by building a system-level metrics catalogue. It employs a multivocal literature review to synthesize process, resource, and product metrics organized under three accountability facets: Responsibility, Auditability, and Redressability, with a GenAI focus. The catalogue translates into concrete governance artifacts, such as roles, committees, training, provenance, and auditing outputs, enabling practical risk management and regulatory alignment. While offering a solid foundation for AI governance, it acknowledges binary scoring limitations and calls for empirical validation and context-aware extensions across industries and jurisdictions.

Abstract

Artificial Intelligence (AI), particularly through the advent of large-scale generative AI (GenAI) models such as Large Language Models (LLMs), has become a transformative element in contemporary technology. While these models have unlocked new possibilities, they simultaneously present significant challenges, such as concerns over data privacy and the propensity to generate misleading or fabricated content. Current frameworks for Responsible AI (RAI) often fall short in providing the granular guidance necessary for tangible application, especially for Accountability-a principle that is pivotal for ensuring transparent and auditable decision-making, bolstering public trust, and meeting increasing regulatory expectations. This study bridges the accountability gap by introducing our effort towards a comprehensive metrics catalogue, formulated through a systematic multivocal literature review (MLR) that integrates findings from both academic and grey literature. Our catalogue delineates process metrics that underpin procedural integrity, resource metrics that provide necessary tools and frameworks, and product metrics that reflect the outputs of AI systems. This tripartite framework is designed to operationalize Accountability in AI, with a special emphasis on addressing the intricacies of GenAI.

Towards a Responsible AI Metrics Catalogue: A Collection of Metrics for AI Accountability

TL;DR

This work tackles the gap between high-level Responsible AI principles and actionable accountability guidance by building a system-level metrics catalogue. It employs a multivocal literature review to synthesize process, resource, and product metrics organized under three accountability facets: Responsibility, Auditability, and Redressability, with a GenAI focus. The catalogue translates into concrete governance artifacts, such as roles, committees, training, provenance, and auditing outputs, enabling practical risk management and regulatory alignment. While offering a solid foundation for AI governance, it acknowledges binary scoring limitations and calls for empirical validation and context-aware extensions across industries and jurisdictions.

Abstract

Artificial Intelligence (AI), particularly through the advent of large-scale generative AI (GenAI) models such as Large Language Models (LLMs), has become a transformative element in contemporary technology. While these models have unlocked new possibilities, they simultaneously present significant challenges, such as concerns over data privacy and the propensity to generate misleading or fabricated content. Current frameworks for Responsible AI (RAI) often fall short in providing the granular guidance necessary for tangible application, especially for Accountability-a principle that is pivotal for ensuring transparent and auditable decision-making, bolstering public trust, and meeting increasing regulatory expectations. This study bridges the accountability gap by introducing our effort towards a comprehensive metrics catalogue, formulated through a systematic multivocal literature review (MLR) that integrates findings from both academic and grey literature. Our catalogue delineates process metrics that underpin procedural integrity, resource metrics that provide necessary tools and frameworks, and product metrics that reflect the outputs of AI systems. This tripartite framework is designed to operationalize Accountability in AI, with a special emphasis on addressing the intricacies of GenAI.
Paper Structure (42 sections, 3 figures, 2 tables)

This paper contains 42 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Challenges
  3. Accountability
  4. The Three Facets of Accountability
  5. Accountability and Transparency
  6. AI Accountability
  7. AI Accountability Challenge
  8. Accountability Challenge in the GenAI Era
  9. Related Work
  10. Methodology
  11. Multivocal Literature Review (MLR)
  12. Systematic Literature Review (SLR)
  13. Grey Literature Review (GLR)
  14. Thematic Coding
  15. Results - Responsibility
  16. ...and 27 more sections

Figures (3)

  • Figure 1: Structure Overview
  • Figure 2: Three Facets of Accountability
  • Figure 3: Methodology Overview