Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evolving AI Risk Management: A Maturity Model based on the NIST AI Risk Management Framework

Ravit Dotan, Borhane Blili-Hamelin, Ravi Madhavan, Jeanna Matthews, Joshua Scarpino

TL;DR

This study tackles the gap between AI ethics principles and actionable risk management by introducing a flexible, RMF-based maturity model for responsible AI governance. It anchors the maturity framework in the NIST AI RMF and provides a flexible questionnaire plus scoring guidelines to assess and guide organizational progress across lifecycle stages and multiple AI systems. The approach supports multiple maturity trajectories, emphasizes sociotechnical harms, and enables evidence-based aggregation by RMF pillars or risk dimensions, aiding benchmarking and improvement. The model aims to operationalize AI risk management in practice, reducing ethics washing and improving adoption of responsible AI practices across sectors.

Abstract

Researchers, government bodies, and organizations have been repeatedly calling for a shift in the responsible AI community from general principles to tangible and operationalizable practices in mitigating the potential sociotechnical harms of AI. Frameworks like the NIST AI RMF embody an emerging consensus on recommended practices in operationalizing sociotechnical harm mitigation. However, private sector organizations currently lag far behind this emerging consensus. Implementation is sporadic and selective at best. At worst, it is ineffective and can risk serving as a misleading veneer of trustworthy processes, providing an appearance of legitimacy to substantively harmful practices. In this paper, we provide a foundation for a framework for evaluating where organizations sit relative to the emerging consensus on sociotechnical harm mitigation best practices: a flexible maturity model based on the NIST AI RMF.

Evolving AI Risk Management: A Maturity Model based on the NIST AI Risk Management Framework

TL;DR

This study tackles the gap between AI ethics principles and actionable risk management by introducing a flexible, RMF-based maturity model for responsible AI governance. It anchors the maturity framework in the NIST AI RMF and provides a flexible questionnaire plus scoring guidelines to assess and guide organizational progress across lifecycle stages and multiple AI systems. The approach supports multiple maturity trajectories, emphasizes sociotechnical harms, and enables evidence-based aggregation by RMF pillars or risk dimensions, aiding benchmarking and improvement. The model aims to operationalize AI risk management in practice, reducing ethics washing and improving adoption of responsible AI practices across sectors.

Abstract

Researchers, government bodies, and organizations have been repeatedly calling for a shift in the responsible AI community from general principles to tangible and operationalizable practices in mitigating the potential sociotechnical harms of AI. Frameworks like the NIST AI RMF embody an emerging consensus on recommended practices in operationalizing sociotechnical harm mitigation. However, private sector organizations currently lag far behind this emerging consensus. Implementation is sporadic and selective at best. At worst, it is ineffective and can risk serving as a misleading veneer of trustworthy processes, providing an appearance of legitimacy to substantively harmful practices. In this paper, we provide a foundation for a framework for evaluating where organizations sit relative to the emerging consensus on sociotechnical harm mitigation best practices: a flexible maturity model based on the NIST AI RMF.
Paper Structure (32 sections, 3 figures)

This paper contains 32 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Background
  3. About maturity models
  4. Maturity models in AI ethics
  5. NIST AI RMF
  6. Design process
  7. Flexible Questionnaire
  8. Flexibility 1: Granularity
  9. Flexibility 2: Life-cycle stage
  10. Flexibility 3: Multiplicity of AI systems
  11. Putting it together
  12. Scoring Guidelines
  13. Scoring Metrics
  14. Coverage of RMF Subcategories
  15. Robustness
  16. ...and 17 more sections

Figures (3)

  • Figure 1: The structure of the questionnaire
  • Figure 2: Illustration of aggregation modes in radio charts: To the left, aggregation by NIST Pillar. To the right, aggregation by responsibility dimension
  • Figure 3: Illustration of maturity progress trajectories. To the left, a bottom-up trajectory. To the right, a top-town trajectory